Table of Contents:
ESS: Latent Class Analysis
Figure 8.1 illustrates the results of the LCA in ESS, where we found the model with three classes to have the best fit. We plot conditional probabilities of specific types of observations on the vertical axis for the three derived classes, where the response options on the horizontal axis are the observations indicating the highest response quality. The three- class LCA solution shows respondents differing strongly with respect to the interviewer observations of respondent behaviors. Respondents assigned to the "Low Quality" class (mean age = 53.2, percent male=43.2, percent lower education = 48.4) frequently ask for clarification, show reluctance, show low effort, and do not understand the questions. These respondents were significantly older and less educated. Our expectation is that this class will provide data of lower quality. Of the 15,816 ESS respondents analyzed, 12% had the highest posterior probability of belonging to this class.
Respondents assigned to the "High Quality" class (mean age=46.8, percent male=46.3, percent lower education=21.9) are more likely to be rated by the interviewers as having understood questions very often, using effort very often, never asking for clarification, and never showing reluctance. These respondents were significantly younger and more highly educated. Roughly 57% of the 15,816 ESS respondents were assigned to this class, which we expect to provide responses of higher quality. The remaining 31% of respondents assigned to the "Moderate Quality" class (mean age=51.5, percent male=43.6, percent lower education =33.9) are expected to provide responses of "moderate" quality, given the profile of this class in Figure 8.1. We note that the three derived classes did not differ in terms of the one observation of the interviewing environment (others present and potentially interfering).
ESS: Class Comparisons on Dependent Variables
We compare the model-based marginal predictions of the means for the data quality indicators and interview length across the three derived classes in Table 8.1. As we expected,
Conditional probabilities of receiving the rating category indicating the highest response quality from the interviewer (ESS).
Comparisons across the Three Derived Quality Classes of Model-Based Marginal Predictions of Means and Probabilities for the Six Response Quality Indicators (ESS)
Note: Different superscripts indicate significant differences of marginal means/probabilities at p < 0.01.
after adjusting for age (which had a significant positive relationship with each dependent variable, except for internal consistency), education (which had a significant negative relationship with each dependent variable, except for internal consistency), and sex (where males tended to be more inconsistent, more acquiescent, and had less missing data) in the multivariable models, respondents assigned to the "Low Quality" class had significantly higher rates of missing data, non-differentiation, extreme answers, and inconsistent answers than respondents in the other two classes. However, inconsistent with expectations based on the literature, respondents assigned to the "Low Quality" class exhibited less acquiescence than respondents assigned to the other two classes. Acquiescence may arise out of deference to the interviewer (Holbrook 2008), so individuals disinterested in the interview and struggling to understand the questions may not have cared about social desirability or pleasing the interviewer.
In terms of mean interview length, measured as seconds per question asked, respondents assigned to the "Moderate Quality" and "Low Quality" classes took more time on average to answer questions than respondents assigned to the "High Quality" class. This is consistent with the class profiles: respondents assigned to these classes needed more clarification, were more reluctant, and generally did not have clear understanding of the questions. Collectively, these ESS results demonstrate the ability of the classes derived based on the post-survey observations of respondent behaviors to distinguish between respondents based on their data quality.
NSFG: Latent Class Analysis
Table 8.2 profiles the seven latent classes derived in the NSFG based on the best-fitting model. We found that 11 of the 22 observations had distributions that varied substantially across the seven derived classes. Five were objective observations of the environment (location of the interview, seating arrangement, distractions due to kids, presence of others, interviewer not happy), and six were subjective observations of respondent behaviors (overall data quality rating, use of headphones in ACASI, respondent attentiveness, respondent tired, respondent not happy, and need for assistance during ACASI). These results suggest that data quality in the NSFG may be a function of both the interviewing environment and specific respondent behaviors.
Table 8.2 shows that nearly two-thirds of the NSFG respondents had higher posterior probabilities of belonging to either class 1 or class 2 than the posterior probabilities of belonging to any other class (like the ESS analysis). Given the characteristics of classes 1 and 2 as described in Table 8.2, we expect these respondents to provide data of relatively
Latent Class Profiles in the NSFG Data
high quality. Respondents assigned in the same way to classes 3 through 5 are expected to provide data of moderate quality, primarily due to child-related distractions and non- conventional settings for the interview (e.g., the respondent's car). Respondents in the last two classes (6 and 7) will likely provide data of questionable quality, for a variety of reasons captured in the interviewer observations and indicated in Table 8.2. Interestingly, while the derived classes are similar in terms of mean age and mean education, the last two classes also have significantly lower percentages of white respondents. Figures A8A.1 and A8A.2 in the online supplemental materials provide additional illustrations of the differences between these seven derived response quality classes.
NSFG: Class Comparisons on Dependent Variables
Table A8A.2 in the online supplemental materials presents comparisons of the model-based marginal predictions of the means and proportions for the NSFG dependent variables across the seven derived response quality classes. Included in Table A8A.2 are indications of which pairwise differences were found to be significant at the 0.05/21=0.002 level (using a Bonferroni correction to account for the 21 pairwise comparisons), suggesting robust differences in the means and proportions across the classes after adjusting for age, race/ ethnicity, and education. The lower-quality classes had interviews that took significantly longer on average after adjusting for the covariates, which is generally consistent with the ESS results. In addition, inconsistencies between the CAPI and ACASI responses on items measured in both modes became significantly more likely in the lower-quality classes, for all four indicators of inconsistent reporting. Figure 8.2 illustrates these trends in the
Differences across the derived quality classes in terms of the marginal predicted probabilities of inconsistent responses in CAPI and ACASI on four NSFG measures.
marginal predicted probabilities of inconsistency of responses across the derived quality classes. Finally, older, African-American, and lower-educated respondents tended to have longer interviews on average, while higher-educated and white respondents tended to provide more consistent reports in CAPI and ACASI.