Data and Preliminary Analysis
We used data from 21 countries* that participated in Round 8 of the European Social Survey, or ESS (Austria, Belgium, Czech Republic, Germany, Estonia, Finland, France, Hungary, Iceland, Ireland, Italy, Lithuania, the Netherlands, Norway, Poland, Portugal, Slovenia, Spain, Sweden, Switzerland, and the United Kingdom).
We computed intra-interviewer correlations for 14 substantive questions from the two rotating modules of the Round 8 ESS questionnaire. Specifically, we selected eight questions from the climate change and energy module (Module D) and six questions from the welfare attitudes module (Module E) (see Online Appendix 22A). Each of the questions used an 11-point response scale, ranging, for example, from 0 = Not at all likely to 11 = Extremely likely, or from 0 = Extremely bad to 11 = Extremely good.
We used the variable EISCED (highest level of education, ES-ISCED) to create the key independent variable for the analysis. This seven-category, harmonized variable is constructed from detailed, country-specific variables and a common ISCED-based coding frame, along with bridging specifications in the ESS. The EISCED variable was recoded into three categories: up to lower secondary education (Level 1: EISCED = 1-2), upper secondary education or advanced vocational education (Level 2: EISCED = 3-5), and tertiary education (Level 3: EISCED = 6-7). This recoding into three categories ensured that a sufficient number of units were available to reliably estimate the IICs in each category, while maintaining a relevant ordering of educational attainment.
The respondent-level control variables included in the models used to calculate the IICs for each education category were as follows: respondent's age (centered around 30) and gender (0 = Female, 1 = Male), whether the language of the interview was the respondent's home language (0 = No, 1 = Yes), and self-reported degree of urbanization of the respondent's domicile, included with dummy variables (1 = Big city [as reference category], 2 = Suburbs or outskirts of big city, 3 = Town or small city, 4 = Country village, 5 = Farm or home in countryside).
The interviewer report, completed by the interviewers at the end of each interview, includes four items describing the respondent's behaviors during the interview. Two of these four assessments are related to the respondent's cognitive abilities: "Did the respondent ask for clarification of any questions?" (RESCLQ) and "Overall, did you feel that the respondent understood the questions?" (RESUNDQ). Both questions were answered on a 5-point scale (1 = Never; 2 = Almost never; 3 = Now and then; 4 = Often; 5 = Very often). These variables can give some indications about whether the interaction proceeded smoothly, and they were used in a preliminary analysis of respondents' understanding of survey questions, described in the following section.
A Preliminary Analysis
A first preliminary step in the analysis consists of the calculation and assessment of the overall IICs. How large are the IICs and are there differences between questions? The basic two-level model with a random intercept and relevant respondent-level control variables
Due to the specific circumstances of the data collection, Russia and Israel are not taken into account.
that we used to obtain an overall IIC for each of the 14 selected substantive questions in each country was specified as follows:
with ец ~ N(0,07),г/0у ~ N^0,a^ and IIC = 2 “ 2.
Figure 22.1 shows the boxplots of the IICs for the 14 questions in each country. The high IICs in some of the countries and the variability across countries are striking. Nine countries (Estonia, Spain, Poland, Ireland, Austria, Czech Republic, Italy, Hungary, and Lithuania) have mean IICs higher than 0.10; there are four countries with a mean IIC in the interval [0.05-0.1] and eight countries have a mean below 0.05. The variability of the IICs within a country tends to be larger for countries with a higher mean IIC. This means that in countries with large interviewer effects, there are also large differences in the interviewer effects for different questions. Overall, the pattern of variability in the IICs suggests that a further evaluation of whether interviewer effects tend to vary across different subgroups of respondents is relevant. As was already argued in the Introduction section, education level seems to be an obvious respondent characteristic to create the subgroups.
Indirect support for the expected link between interviewer effects and respondents' education level is derived from the interviewers' reports on respondents' understanding of survey questions. A higher frequency of asking for clarification and explaining the questions is expected for lower-educated respondents than for higher-educated respondents. Inadequate understanding and requests for clarification intensify the interaction between the respondent and the interviewer, resulting in a higher risk of interviewer effects.
A descriptive analysis of the mean frequency of understanding questions and asking for clarification (Table 22.1) shows that the patterns of the means are consistent with
Boxplots of the IICs for the 14 questions by country.
Mean Frequency of Respondents Understanding the Questions and Asking for Clarification by Education Level
Note: 1 = Never; 2 = Almost never; 3 = Now and then; 4 = Often; 5 = Very often, i.e., higher values indicate a higher frequency.
these expectations. In all countries, the mean score for asking for clarification is higher for lower-educated respondents than for higher-educated respondents, indicating that lower- educated respondents tend to ask more frequently for clarification. The mean scores for understanding the questions are high for all three education levels, but in most countries the mean score is highest for higher-educated respondents, indicating that respondents in this group experience fewer comprehension problems. These results support the proposition that lower-educated respondents experience more difficulties when answering the questions asked during a face-to-face interview.