Desktop version

Home arrow Mathematics

  • Increase font
  • Decrease font


<<   CONTENTS   >>

Data and Methods

A meta-analysis consists of five steps: (1) a comprehensive literature search; (2) checking the eligibility of studies found; (3) coding of relevant data; (4) calculation of training effect sizes; and (5) analysis of variables that moderate effect size (Borenstein et al. 2009; Lipsey and Wilson 2001). Additional information on each of the steps can be found in the online supplementary materials in the Appendix. To be included in our meta-analysis, studies must have employed an experimental design. We accepted both designs that randomly assigned interviewers to treatments versus control conditions and those that used measurements on experimental pretest-posttest designs. For both types of training, it was essential that the control group received either no or only an introductory briefing. The search was limited to literature in English; over 2,000 results had to be excluded because the broad search terms generated literature related to job interviews, linguistic interviews, cognitive and clinical interviews of victims and witnesses, and studies without an experimental design. Only 14 eligible publications were retrieved, however most of the publications reported more than one experiment or effect size. For a search overview, see Figure 4.1. The most common indicator of data quality was the effect of interviewer training on the response rate (24 effect sizes nested in 11 manuscripts), followed by response accuracy (21 effect sizes nested in 2 manuscripts); correct recording of the response (14 effect sizes all from 1 manuscript); item nonresponse (12 effect sizes nested in 3 manuscripts); reading of questions exactly as worded (12 effect sizes all from 1 manuscript); correct probing (6 effect sizes all from 1 manuscript); and correct item administration (4 effect sizes nested in 2 manuscripts). To account for the fact that several effect sizes are nested within studies, we estimated multilevel models for robustness checks. For correct reading, probing, and recording, where we could identify only one study (Fowler and Mangione 1990), we discuss the outcomes qualitatively as well as response accuracy where the true values were unknown.

Results

What Is the Effect of Interviewer Training on Data Quality?

Refusal avoidance training and survey response rates (Q1 in Table 4.1). Figure 4.2 shows a forest plot summarizing the study-level differences in response rates between trained and

FIGURE 4.1

The literature search process.

untrained interviewers. The x-axis presents the estimated differences in data quality between trained and untrained interviewers. Positive values mean better data quality - for this outcome, higher response rates - for trained interviewers and all confidence intervals (CIs) that do not cross the zero line are significantly different from zero. The у-axis shows all included studies, and each point represents their effect sizes and confidence intervals. The last line of each quality measure with the title "RE model" shows the sampling error weighted mean effect size under the meta-analytic random effects assumption. The effect size distribution in the forest plot indicates that most response rate comparisons show that trained interviewers achieved higher response rates than interviewers without specific refusal aversion training. Surprisingly, there were quite a few zero findings. The sampling error weighted mean effect size estimate, calculated across all 24 effect sizes assuming random effects, was 0.05 (95% Cl = 0.0/0.1). This result shows that the response rates achieved by trained interviewers were, on average, five percentage points higher (with a confidence

FIGURE 4.2

Forest plots for data quality indicators: trained vs. untrained interviewers.

interval from no effect to 11 percentage points) than those achieved by untrained interviewers. Our first research question (Ql) can therefore be answered in the affirmative[1] [2] [2] However, the improvement of five percentage points (Cl 0,11) for nonresponse rates is surprising and indicates that interviewer training has a moderate impact on response rates.

Questionnaire administration training and interviewer effects (Q2 in Table 4.1). Looking next at item nonresponse, our results suggest that trained interviewers achieved significantly higher data quality than untrained interviewers (see Figure 4.2). In particular, we found that, across the three studies, trained interviewers had lower item nonresponse rates than untrained interviewers (4%; 95% Cl = -0.07/-0.02).+

Across two studies (Benson and Powell 2015; Guest 1954), trained interviewers were more likely to do a set of tasks required of interviewers. In particular, for Benson and Powell (2015), trained interviewers were more likely to administer different question types and probe nondirectively and not emphasize certain response options than untrained interviewers. For Guest (1954), trained interviewers were more likely to hold a discussion of interviewing in general and how to sample and the handling of different question types, than untrained interviewers.

We were able to identify two studies (Canned, Oksenberg, and Converse 1977; Miller and Canned 1982) that evaluated whether specific experimental interviewing procedures (i.e., commitment statements, instructions, and scripted feedback; see Chapter 1 in this volume) increased the accuracy of responses. In both studies, the true values were unknown and increased accuracy was inferred through a change in the mean of the responses; thus, we summarize the results qualitatively. Both studies found positive effects of interviewing procedures compared to interviews without these features. In Miller and Canned (1982), six out of ten socially desirable questions where underreporting was expected had higher average reports in the groups where interviewers used advanced techniques. Canned, Oksenberg, and Converse (1977) recorded more accurate responses (e.g., respondents checked their documents to report on the last doctor visit) to 9 out of 11 variables with the advanced techniques.

We could only identify one study (Fowler and Mangione 1990) that examined the effect of training for the other three outcomes (correct reading, probing, and recording). Fowler and Mangione (1990) tested various interviewer training durations and training methods. The control group always received basic training. This basic training consisted of a two- hour lecture on interviewing techniques and one demonstration interview. The authors conducted a total of 12 experiments on correct probing, which differed in training content and duration, eight of which showed no significant difference between the experimental and control groups. In the remaining four experimental groups, Fowler and Mangione (1990) found a 20-50% increase in the rate of correct probing. The test groups with the longest trainings of ten days had the biggest improvements (40%) in probing, the difference between the five-day trainings and the ten-day trainings were in more time for practicing. For the correct reading of questions, significant improvements of up to 50% could be observed for half of the experiments depending on training content and training duration. Five-day trainings including lectures, demonstrations, a movie, and practicing performed best. For the correct recording of the questions, no improvement was found in nine experiments; the remaining four experiments showed 20-30% more correct recording. The best results were again revealed after the five-day trainings.

Due to the small number of studies, conclusions from these analyses are necessarily suggestive rather than definitive. This overall picture suggests that interviewer refusal avoidance training reduces unit nonresponse rates. Thus, Q1 can be answered in the affirmative. For the other data quality indicators related to measurement error, there is much more limited evidence for improvements. Thus, Q2 requires further investigation.

Effect size heterogeneity (Q3 in Table 4.1). The heterogeneity of training approaches we observed across these studies prompted us to ask whether effect size distributions were heterogeneous (Q3), which would result in further moderator analysis. This question can be answered in the affirmative for the studies on refusal avoidance training (p < .05), assuming random effects (see Appendix Table A6). To examine whether - and, if so, which - interviewer training features influenced the effect of interviewer training on response rates, we conducted a moderator analysis.

  • [1] * Using a multilevel model, which accounts for nesting of several effect sizes in one manuscript, confirms thisfinding (4.5 percentage points difference (Cl: -0.07/0.01), with almost 15% of the variance explained on thestudy level).
  • [2] The multilevel model showed for item nonresponse a difference of 2.7% with 63% variance on the study level.
  • [3] The multilevel model showed for item nonresponse a difference of 2.7% with 63% variance on the study level.
 
<<   CONTENTS   >>

Related topics