Desktop version

Home arrow Mathematics

  • Increase font
  • Decrease font

<<   CONTENTS   >>

Dependent Variables in the ESS

In the ESS, we computed six dependent variables, including indirect measures of data quality, measures of response styles, and proxies of the respondent's cognitive response process. The first indirect data quality indicator captured item nonresponse, measuring the percentage of survey variables where respondents indicated "refusal," "don't know," or "no answer." The second indirect data quality indicator captured a lack of differentiated answers. We chose from the ESS several battery questions (see the Online Appendix), each of which included several attitudinal items using the same response scale. We only used item batteries that were comprised of at least four items. We defined a response pattern to an item battery as non-differentiating if a respondent chose the same answer throughout the battery. The overall indicator was the percentage of batteries where non-differentiation/ straight-lining occurred.

For the third dependent variable, we assessed the number of extreme answers as a measure of a sub-optimal response style that can bias scale scores and affect scale reliability, in addition to affecting relationships between variables (Alwin and Krosnick 1991;

Diamantopoulos, Reynolds, and Simintiras 2006; Leventhal 2018). We considered the same survey items used to measure non-differentiation and simply calculated the percentage of questions answered with a 1 or 5 (on the five-point scale), or with a 1 or 10 (for ten-point scale items). ESS respondents could therefore have high measures of non-differentiation and extreme answers simultaneously, but this was rare; the correlation of these two dependent variables was only 0.06. For 23 attitudinal items within the ESS for which affirmative answers were possible, we computed the percentage of items where the respondent agreed or supported the affirmative as a fourth indirect indicator of data quality (Van Vaerenbergh and Thomas 2013). This type of acquiescent response style is a source of measurement error that, being a systematic behavior on the part of the respondents, can lead to bias in estimation (Reynolds and Smith 2010; Roberts 2016), and has also been studied as a type of socially desirable responding (Holbrook 2008). We did not include items bearing any reference to factual events as it is less likely that respondents affirm a factual question falsely.

For our fifth dependent variable, we constructed a measure of internal consistency in the answers provided by the respondent based on the following two items: "All laws should be strictly obeyed" and "Doing the right thing sometimes means breaking the law." Both items had to be answered using a five-point agree-disagree scale. The idea behind these items is the same, except that the statements are reversely worded. Agreeing to both statements can therefore be regarded as a sign that the cognitive process necessary to provide an optimal answer is incompletely or superficially performed. The resulting variable is a dummy variable that equals 1 if the answers to the items were inconsistent. Finally, interview length, operationalized in the ESS as the number of seconds spent per question asked, was also included in the analyses as an indirect measure of data quality, where longer surveys tend to yield data of lower quality (Galesic and Bosnjak 2009; Herzog and Bachman 1981). To deal with the extreme skewness of some of the variables, we applied natural log transformations to the item nonresponse rate, the extreme answer rate, and interview length. Collectively, these six measures tended to capture different indirect aspects of data quality and the response process, having pairwise correlations that were no greater than

0.28 in absolute value (see Table Table A8A.2 in the Online Appendix).

Dependent Variables in the NSFG

In the NSFG, our dependent variables measuring indirect indicators of data quality included one paradata variable (total interview length in minutes; Galesic and Bosnjak 2009; Herzog and Bachman 1981) and four proxy indicators of measurement error. These proxy indicators were binary indicators of inconsistent responses (1 = different responses, 0 = identical responses) for four survey variables that were measured in both CAPI and ACASI: number of sexual partners in the past year (females), number of sexual partners in the past year (males), number of live births to date (females), and total number of pregnancies fathered (males).

Analytic Approach

We first applied latent class analysis (e.g., Kreuter, Yan, and Tourangeau 2008) to the interviewer observations. Briefly, this multivariable modeling approach takes as input multiple categorical variables (e.g., the post-survey interviewer observations from each survey listed above) that an analyst believes indicate some latent (unmeasured) categorical trait (e.g., overall response quality). The analyst specifies a hypothesized number of categories, or classes, for the latent categorical trait, and the estimation procedure results in predicted probabilities of membership in each class for each case in the data set (e.g., survey respondent). In addition, the estimation procedure generates conditional probabilities of each category on the input items, depending on the predicted class membership; this allows analysts to profile the latent classes in terms of distributions on the observed categorical measures. One can evaluate the fits of competing latent class models with different counts of hypothesized categories for the latent trait and select the model that provides the best fit to the observed data.

For the purposes of the present study, we identified the best-fitting latent class model in terms of the number of latent response quality classes of respondents (given the input interviewer observations from each survey); we considered models with between two and seven classes. Given the best-fitting model for one of the two data sets, we then computed predicted probabilities of belonging to each class for the respondents. We also computed predicted probabilities for each of the values of the input observations as a function of class membership based on the best-fitting model. We assigned each survey respondent to the class for which they had the highest predicted probability (as one possible "modal" method of predicting class membership), and then profiled the resulting classes (e.g., high quality, low quality, etc.) based on the predicted probabilities for the values of each interviewer observation for each class.

We used PROC LCA in SAS (Lanza et al. 2007) to fit all latent class models, including the CLUSTER option to account for potential correlations in the observed data due to the interviewers. We compared model fit statistics (including the log-likelihood, g-squared, AIC, BIC, and adjusted BIC) and entropy (where higher entropy values indicate better class separation) across the competing models. In the NSFG, we found that the model with seven latent classes had by far the best fit in terms of minimizing all of the model fit criteria (see Table A8A.1 in the Online Appendix) and had an entropy estimate of 0.89, indicating good separation of the classes. For the ESS, we found that the best-fitting model had three latent classes when using a similar approach (fitting models with 4+ classes was not possible with only five input variables).

Next, we compared the predicted marginal means of the dependent variables between the derived latent response quality classes using PROC SURVEYREG or PROC SURVEYLOGISTIC in SAS (again accounting for clustering by interviewer), depending on the type of dependent variable. In these models, we adjusted for age, race/ethnicity (only measured in the NSFG), gender (only relevant in the ESS, since the NSFG measures were gender-specific), and education, to allow for possible associations of both the interviewer observations and the dependent variables with these observable socio-demographic characteristics and isolate the relationships of the response quality classes with the dependent variables when adjusting for these characteristics. We then performed pairwise comparisons of the model-based marginal predictions of the means and proportions for each of the dependent variables across the derived classes.

We used the distal LCA macro in SAS (Lanza, Tan, and Bray 2013) to assess the robustness of our findings about differences between the classes to the assumption that the latent response quality classes for each respondent were known with certainty (i.e., using predicted probabilities of class membership, rather than fixed assignments based on the "modal" approach). This macro currently does not have the ability to account for interviewer clustering, so we focused on general patterns of differences in means and proportions between the classes in these supplemental analyses. In general, we found that our primary results did not change when allowing for uncertainty in the predicted class membership.

<<   CONTENTS   >>

Related topics