Desktop version

Home arrow Mathematics

  • Increase font
  • Decrease font


<<   CONTENTS   >>

Methods

Multilevel analysis is now a standard approach for the analysis of interviewer effects on nonresponse (Blom, et al. 2011). We started our analysis by fitting null models, more specifically three-level regressions (PIAAC) and two-level regressions (GIP 2012, SHARE, and GIP 2014), to identify the amount of raw variance at the interviewer level (Hox 2010). Due to differences in sampling methods and gross sample sizes, the PIAAC survey allocated at least two primary sampling units (PSUs) to each interviewer for most of the sample, while interviewers in the GIP 2012 survey, the GIP 2014 survey, and the SHARE survey worked in one PSU only. For this reason, three-level logistic regression models with sample units nested within PSUs and PSUs nested within interviewers were estimated for PIAAC, while two-level logistic regression models with sample units nested within interviewers were estimated for GIP 2012, GIP 2014, and SHARE.

The four surveys were implemented without an interpenetrated sample design that would enable us to separate interviewer effects from area effects. Studies of interviewer effects in surveys employing face-to-face recruitment often suffer from this problem. Following the guidance of Schaeffer, Dykema, and Maynard (2010), we specify multilevel models that include fixed effects for variables describing the sample compositions of the interviewers' assignments for all models, including the size of the sample point in terms of number of inhabitants, the region with four categories (north, south, west, east), the share of Germans, the share of single-person households, and the share of unemployed persons in the immediate neighborhood of the sampled addresses.

Taking the random effects at the interviewer level into account, we next specified a model for each survey outcome (contact and cooperation) that included all available interviewer characteristics from the interviewer questionnaire described above, in addition to the sample composition control variables. Models were estimated using the xtme logit procedure in Stata 13. We tested odds ratios for significance using a two-sided Wald test, while interviewer variance components were tested against zero using an appropriate likelihood-ratio test (Rabe-Hesketh and Skrondal 2012).

Results

Our results show that for GIP 2012, PIAAC, and GIP 2014, around 20% of the variance in a sampled person's propensity to be successfully contacted is due to interviewer effects (GIP 2012: 18.5%, PIAAC: 20.7%, GIP 2014: 17.2%). The percentage of variance in a sampled person's propensity to be successfully contacted that can be explained by interviewer effects is much higher for SHARE (60.6%). These notable interviewer effects on contact in SHARE are likely arising from the fact that most of the interviewers in the SHARE sample have a contact rate of 100%, and only some interviewers are found to be "outliers" with a very low contact rate. Thus, we focus on explaining interviewer effects on contact for GIP 2012, PIAAC, and GIP 2014. (Regression tables are presented in Online Appendix 14E.)

In contrast, when examining a sampled person's propensity to participate in each of the respective surveys, the fraction of the raw variance due to interviewer effects is rather small for both PIAAC (2.1%) and SHARE (5.2%). The percentages of variance in propensity to participate due to interviewer effects for GIP 2012 and GIP 2014 are slightly higher

(GIP 2012: 12.8%, GIP 2014: 17.2%). As the fraction of the variance in a sample person's willingness to participate in PIAAC due to interviewer effects is so small in PIAAC, we do not conduct further analysis using interviewer characteristics. Thus, we focus on explaining interviewer effects on cooperation for GIP 2012, SHARE, and GIP 2014 (see regression tables in Online Appendix 14E).

The results from fitting the three-level logistic regression model for PIAAC and the two- level logistic regression models for GIP 2012 and GIP 2014 that added the fixed effects of interviewer characteristics to explain interviewer effects on a sample person's propensity to be successfully contacted are presented in Figure 14.1. The results from the models for a sampled person's propensity to participate in the survey are presented in Figure 14.2. In the plot, the markers are odds ratios, and the horizontal lines are confidence intervals.

Looking at Figure 14.1, it becomes obvious that none of the available interviewer characteristics have a significant relationship with a sampled person's propensity to be successfully contacted in each of the four surveys (see Online Appendix 14E).

For GIP 2012, we find that sample persons have a significantly higher propensity to be successfully contacted when interviewers report that they do stick to the interviewer instructions during the implementation of the questionnaire. The interviewer variance explained by including all interviewers' characteristics in one model is about 19%.

For PIAAC, we find that sample persons have a significantly lower propensity to be successfully contacted when interviewers report a higher tendency to tailor questions in order to shorten the questionnaire. Persons have a significantly higher propensity to be successfully contacted when interviewers report that they either deviate from standardized interviewing techniques to help respondents, such as speaking slower or speaking the dialect of the respondent, or report they stick to interviewer instructions. The latter two significant results are contradictory, as deviating from standardized techniques and sticking to instructions are seemingly opposite types of interviewer behaviors. Also, sample persons have a significantly higher propensity to be successfully contacted when interviewers say that refusals should not be accepted when respondents are reluctant. The interviewer variance explained by including fixed effects of all the interviewers' characteristics in one model is about 56%.

For GIP 2014, we find that sample persons have a significantly lower propensity to be successfully contacted when interviewers report a lower agreeableness towards the voluntariness of participation. The interviewer variance explained by including all interviewers' characteristics in one model is about 30%.

In Figure 14.2, we present estimated odds ratios from the two-level logistic regression models for cooperation in GIP 2012, SHARE, and GIP 2014. Similar to the results for the contact models, we do not find any interviewer characteristic that has a significant relationship with cooperation in each of the four surveys in our study.

For GIP 2012, our results indicated a significantly higher propensity to cooperate when female interviewers tried to gain the cooperation of sampled persons. Furthermore, we found a significant positive effect of tailoring content: sampled persons approached by interviewers who report that they tailor content less frequently have a higher propensity to cooperate. Sampled persons had a significantly lower propensity to participate when interviewers report a higher tendency to tailor questions in order to shorten the questionnaire. Furthermore, sampled persons' propensity to cooperate is significantly lower when interviewers report being concerned about data protection issues themselves. The interviewer variance explained by including the fixed effects of all the interviewers' characteristics in one model is about 45%.

FIGURE 14.1

Estimated odds ratios for predictors of successful contact, multilevel logistic regression, across surveys. Note: Parameter estimates with 95% confidence intervals from three-level logistic regression for PIAAC and from two-level logistic regression for GIP 2012 and GIP 2014 for the dependent variable of successful contact. Significant = p < 0.05. Model controls for sample composition characteristics (share of Germans, share of single-person households, share of unemployed persons in the immediate neighborhood of the sampled household, PSU size, and region). Reference age: < 45 years. Reference gender: the interviewer is male. Reference education: interviewer has no Abitur (А-levels). Reference employment status: interviewer is not full- or part-time employed. Reference experience: < 5 years of experience. Reference working hours: < 15 hours per week. Number of interviewers GIP 2012 = 131. Number of interviewers PIAAC = 115. Number of interviewers GIP 2014 = 141. Number of F’SUs PIAAC = 251. Number of sample persons GIP 2012 = 3,898. Number of sample persons PIAAC = 7,989. Number of sample persons GIP 2014 = 6,253. PIAAC = Programme for the International Assessment of Adult Competencies. GIP = German Internet Panel. The full models can be found in Online Appendix 14E.

FIGURE 14.2

Estimated odds ratios for predictors of successful cooperation, multilevel logistic regression, across surveys. Parameter estimates with 95% confidence intervals from two-level logistic regression for GIP 2012, SHARE, and GIP 2014 for the dependent variable of successful cooperation. Significant = p < 0.05. Model control for sample composition characteristics (share of Germans, share of single-person households, share of unemployed persons in the immediate neighborhood of the sampled household, PSU size, and region). Reference age: < 45 years. Reference gender: the interviewer is male. Reference education: interviewer has no Abitur (A-levels). Reference employment status: interviewer is not full- or part-time employed. Reference experience: < 5 years of experience. Reference working hours: < 15 hours per week. Number of interviewers GIP 2012 = 131. Number of interviewers SHARE = 142. Number of interviewers GIP 2014 = 141. Number of sample persons GIP 2012 = 3,038. Number of sample persons SHARE = 7,058. Number of sample persons GIP 2014 = 5,767. GIP = German Internet Panel. SHARE = Survey of Health, Ageing, and Retirement in Europe. The full models can be found in Online Appendix 14E.

For SHARE, we found three interviewer characteristics that had significant relationships with cooperation propensity, and all three were indicators measuring how to successfully achieve response. Persons had a significantly higher propensity to participate in SHARE when interviewers reported fewer acceptances of respondents' refusals, when interviewers reported higher agreeableness towards the voluntariness of participation, and when interviewers reported higher agreeableness towards the statement that people will cooperate when they are contacted at the right time. The interviewer variance explained by including fixed effects of all the interviewers' characteristics in one model is about 36%.

For GIP 2014, sampled persons had a significantly lower propensity to participate when interviewers with an Abitur (А-levels) education tried to gain their cooperation, compared to interviewers who have a lower educational degree. In addition, sampled persons approached by interviewers who reported that they tailored less frequently had a lower propensity to cooperate, and sampled persons approached by interviewers who reported a higher agreeableness towards the statement that people will cooperate when they are contacted at the right time had a higher propensity to cooperate. Also, sampled persons' propensity to cooperate is significantly higher when they are approached by interviewers working more than 30 hours per week, compared to interviewers working less than 30 hours per week. The interviewer variance explained by including the fixed effects of all the interviewers' characteristics in one model is about 35%.

 
<<   CONTENTS   >>

Related topics