Desktop version

Home arrow Mathematics

  • Increase font
  • Decrease font


<<   CONTENTS   >>

Data and Methods

Table of Contents:

Data for this study are from the Voices Heard computer-assisted telephone interview (CATI) survey, which was designed to measure perceptions of barriers and facilitators to participating in medical research studies that collect biomarkers (e.g., saliva and blood) among respondents from various racial and ethnic groups (White, Black, Latino, and American Indian). We employed a quota sampling strategy because screening to identify members in non-White groups would have been prohibitively expensive. The quota sample consisted primarily of volunteers but also used a targeted list of names provided by a commercial vendor (see Online Appendix 18C for more detail). Interviewers conducted 410 usable interviews (in English only) with an average length of 25.21 minutes between October 2013 and March 2014. Respondents received a $20 cash incentive. The 96 questions included in the survey asked about: likelihood to participate in medical research based on the type of study (e.g., to collect tissue) and characteristics of requestor (e.g., "a member of your community"); things medical researchers do to encourage participation (e.g., provide results); concerns about participating in medical research; attitudes toward medical researchers; health status, health-related quality of life, health behaviors and conditions, and health care use; knowledge of research procedures; and social and demographic characteristics.

Measures

Dependent variable. RTs were collected by the CATI computer software as the amount of time (in seconds) spent on each question (mean 13.22 seconds, standard deviation 8.96, range 1-110). Values were top- and bottom-coded at the 99th and 1st percentiles within each item and log-transformed to correct for outliers and skew (Yan and Tourangeau 2008).

Individual question characteristics. Research assistants coded the previously identified individual question characteristics (Hla-Hll in Table 18.1) under the direction of the authors; no interrater reliability statistics were calculated, but codes were verified by the first author. Descriptive statistics for question characteristics are provided in the first column of Table 18.2.

Established tools for evaluating questions. We measured readability using the Flesch- Kincaid grade level. A higher level indicates the question's text is more difficult to read. For QUAID, we tallied the number of problems flagged by the online tool across five comprehension difficulty categories. QAS was coded by a member of the research team and operationalized as a composite sum of the number of problems identified out of 27 possible problems. SQP was coded by an undergraduate research assistant using SQP's online documentation. We use SQP's "quality estimate" (the product of a question's estimated reliability and validity) (see Online Appendix 18B).

Interviewer and respondent characteristics. The key interviewer characteristic of interest is within-study experience (number of interviews the interviewer completed up to the current interview). Other interviewer characteristics included as controls are: race (White, non-White [very few interviewers were Black, Latino, or Asian]), gender, age, and prior interviewing experience (less than one year or one year or more). Respondent characteristics included as controls are: race/ethnicity (Black, Latino, American Indian, and White), gender, age, and education (high school education or less, some college, and college or more). The last two characteristics are used in prior studies to examine or control for factors associated with response processing and cognitive ability (see Online Appendix 18D, Table A18.D1).

Analytic Strategy

The analytic sample includes 410 respondents asked 95 or 96* questions by one of 24 interviewers, yielding 39,052 question-answer sequences, which are the unit of analysis. We use cross-classified random-effects linear regression models to predict the log-transformed RTs using Stata 15.1. We use the mixed command with restricted maximum likelihood (retnl) to analyze the data with a variance structure that uses crossed random effects to account for the fact that RT for each question is measured for each respondent and interviewer, and

One question was a follow-up to a filter question that was not asked if respondents answered "yes" to the filter question.

Question Characteristics

Descriptive Statistics

Regression

Mean or Percent

Std.

Dev.

Min.

Max.

Coef.

Std.

Err.

Number of words

30.47

16.17

5.00

75.00

0.018

0.003

***

Question order

48.50

27.86

1.00

96.00

-0.002

0.002

Question type

Event or behavior (reference category)

57.3%

Subjective

28.1%

0.211

0.147

Demographic

14.6%

0.231

0.155

Question form Yes/no (reference category)

30.2%

Nominal

8.3%

0.208

0.123

Discrete value

2.1%

0.328

0.188

Bipolar ordinal

16.7%

1.067

0.170

***

Unipolar ordinal

42.7%

0.600

0.138

***

Definition in the question (vs. not)

5.2%

0.079

0.159

List-item question (vs. not)

35.4%

0.027

0.063

Sensitive question (vs. not)

10.4%

0.073

0.088

Race-related question (vs. not)

9.4%

-0.017

0.100

Battery structure First in battery

9.4%

0.102

0.114

Later in battery (reference category)

44.8%

First in series

6.3%

0.275

0.127

*

Later in series

31.3%

0.165

0.105

Stand-alone

8.3%

0.242

0.139

Emphasis in the question (vs. not)

19.8%

-0.316

0.102

Jt*

Interviewer instructions (vs. not)

9.4%

0.201

0.112

Parenthetical phrases (vs. not)

34.4%

-0.336

0.082

***

Flesch-Kincaid grade level

12.22

5.16

0.00

22.10

0.018

0.008

*

QUAID problem score

4.38

2.30

1.00

12.00

0.012

0.014

QAS problem score

1.00

1.02

0.00

4.00

-0.033

0.045

SQP quality score

0.50

0.05

0.44

0.67

-0.319

0.720

Intercept

1.166

0.443

**

Random-effects parameters

Interviewer-level variance

0.003

0.001

*

Question-level variance

0.045

0.007

***

Respondent-level variance

0.012

0.001

***

Residual variance

0.085

0.001

***

Wald chi-square

693.83

(df 35)

***

Log-restricted likelihood

-8,268.60

Notes: Std. Dev. = standard deviation, Min. = minimum, Max. = maximum, Coef. = coefficient, Std. Err. = standard error. Descriptive statistics are calculated at the level of the question (N=96) for question characteristics. Regression analysis is conducted at the level of the question-answer sequence (N=39,052). Regression model also controls for respondent (race/ethnicity, gender, age, and education; N = 410) and interviewer characteristics (race/ethnicity, gender, age, prior interviewing experience, and study-specific experience; N = 24).

*p < 0.05, *‘p < 0.01, *‘*p < 0.001.

respondents are nested within interviewers. The base model predicting RT i for question/!, respondent j2, and interviewer к is ln(Response time).,, . = Д, + «„ +uh +е«А,й». In this model, M„ ~N(0,2)), vk ~ N(0,cr2), and eiiilij2)k ~ N(0,cr2)).

The full model predicting RT includes a series of fixed effects for questions, respondents, and interviewers:

Because RTs are (natural) log-transformed, the coefficients can be interpreted in terms of percentage change, such that RTs change by 100 [1] [exp(P) - 1] percent for a one- unit increase in the independent variable, holding all other variables in the model constant.

  • [1] The discrete-value questions ask respondents to report numerical answers: year of birth and number ofdays they drank alcohol in the past month. RTs are lower for all question forms compared to bipolar ordinalquestions (p<.001) and lower for nominal questions compared to unipolar ordinal questions (p<.01) (notshown).
 
<<   CONTENTS   >>

Related topics