Desktop version

Home arrow Mathematics

  • Increase font
  • Decrease font


<<   CONTENTS   >>

Discussion

This chapter presented an application of LCA to multiple post-survey interviewer observations from two major surveys (the NSFG and the ESS). The results suggest that applying this type of approach to post-survey interviewer observations of the survey response process can produce meaningful classes of respondents that vary in terms of the quality of their reporting. This approach can also identify those observations (whether they are observations of the environment or respondent behaviors) that are the most important for defining the response quality classes, in that they have distributions that vary substantially across the derived classes. In the NSFG, both types of observations contributed to defining the quality classes, suggesting that both respondent behaviors and the interviewing environment can affect response quality; this makes sense given the sensitive subject matter about sexual health. In the ESS, only the observations of respondent behaviors were found to vary across the derived quality classes.

If a latent class model fitted to the observations in a given survey has a reasonably good fit, and meaningful distinctions between the response quality classes emerge, one could use this technique to compute a single categorical variable that contains predicted response quality classes for all respondents in a survey data set. Secondary analysts could then use this variable to adjust for response quality in their analyses and determine whether estimates of interest change substantially across the response quality classes. Importantly, this type of quality indicator should not be treated as a causal predictor of poor data quality, given that the observations are endogenous to the survey response process. Rather, analysts should view this indicator as a broad summary of the observed interaction and use it only to adjust estimates of interest for questionable behaviors and difficulties encountered during the interview itself.

The idea of providing data users with this type of quality indicator needs more consideration by survey organizations in the future to justify the resources dedicated to the continued collection of these observations. In short, the observations need to be a clearly defined part of a larger analytic plan. More generally, the ability of these observations to broadly classify respondents in terms of the quality of their responses suggests that interviewers can accurately detect undesirable response behaviors (non-differentiation, item-missing data, inconsistency, etc.). Interviewers could therefore be trained to intervene during the interview when these behaviors emerge, and provide encouragement, motivation, etc., to prevent lower-quality responses in real time.

Finally, we did not specifically address the possibility that the post-survey interviewer observations themselves may be of reduced quality. Past studies have presented evidence of significant interviewer variance in the distributions of these types of observations (e.g., Kirchner, Olson, and Smyth 2018; O'Muircheartaigh and Campanelli 1998; West and Peytcheva 2014). Future research should investigate this issue in more detail and focus on sources of any unexplained variance among interviewers in terms of distributions on these observations.

Acknowledgments

We acknowledge financial support for this work from a methodological research modification to the contract between the University of Michigan and the National Center for Health Statistics (NCHS) that supports the National Survey of Family Growth. The National Survey of Family Growth is conducted by the Centers for Disease Control and Prevention's (CDC's) National Center for Health Statistics (NCHS), under contract # 200- 2010-33976 with University of Michigan's Institute for Social Research with funding from several agencies of the U.S. Department of Health and Human Services, including CDC/ NCHS, the National Institute of Child Health and Human Development (NICHD), the Office of Population Affairs (OPA), and others listed on the NSFG webpage (see www.cdc. gov/nchs/nsfg/). The views expressed here do not represent those of NCHS nor the other funding agencies.

 
<<   CONTENTS   >>