Measuring strategic processing through concurrent and task-specific self-reports
Self-reports of strategic processing can involve on-line or off-line reporting. On-line approaches refer to measurements taken concurrent to task performance, such as having learners think aloud to create a verbal protocol that subsequently can be analyzed by the researchers. Off-line approaches refer to self-report inventories or interviews administered before or after task performance. In this chapter, we consider on-line measures in the form of concurrent thinking aloud and off-line measures in the form of task-specific self-reports collected immediately after task performance.
Verbal Protocol Analysis
In research on text-based learning and comprehension, individuals can be instructed to think aloud as they read one or more texts (Ericsson & Simon, 1993; Pressley & Afflerbach, 1995; Trabasso & Magliano, 1996). This means that readers are asked to report whatever thoughts come to mind as they read, and the intent is to have them report only those thoughts that are immediately accessible and reportable in language, and therefore represent the contents of working memory (Ericsson & Simon, 1993). This activity can be considered to reflect an effortful search for meaning and, as such, to provide a window on strategic processing (Trabasso & Magliano, 1996). More specifically, verbal protocols resulting from concurrent thinking aloud are considered a valid measure of the metacognitive states that arise during reading and the strategies that respond to these states (Pressley & Afflerbach, 1995), in particular inference processes that support the construction of a mental model (Trabasso & Magliano, 1996). In this section, we describe two approaches to collecting verbal protocol data and discuss available evidence that these approaches are valid measures of comprehension strategies.
Approaches to Collecting Verbal Protocols. One approach is to allow individuals to self-select when and where they think aloud while reading (Goldman, Braasch, Wiley, Graesser, & Brodowinska, 2012; Pressley & Afflerbach, 1995). This approach is viewed as sensitive to metacognitive states that arise during reading and strategies readers employ in response to them (e.g., experiencing confusion after reading a section and then rereading that section in response to that metacognitive state; Pressley & Afflerbach, 1995). That is, readers likely report thoughts that reflect an effortful search for meaning, especially when experiencing challenges understanding the texts.
An alternative approach is to have readers report their thoughts after pre-selected locations, which could be after every sentence, paragraph, or section (Trabasso & Magliano, 1996), or after theoretically determined locations (Kaakinen & Hydna, 2005; Magliano & Millis, 2003). This approach can be construed as a retrospective protocol because participants are asked to reflect upon cognitive states that have recently occurred but likely remain in working memory (Ericsson & Simon, 1993). Typically, the intent is to reveal thoughts that pertain to mental model construction, in particular (Magliano, 1999; Trabasso & Magliano, 1996). Thus, this approach aims to reveal the products of knowledge activation and inference generation to integrate text constituents in the mental model. This may involve establishing how text constituents are semantically connected (e.g., cause and effect, claim-evidence, and contrastive relationships; Magliano, Trabasso, Graesser, 1999; Ray & Magliano, 2015) or integrating text constituents with background knowledge (Todaro, Magliano, Millis, McNamara, & Kurby, 2008). This approach is suitable for gaining access to processes involved in mental model construction for at least two reasons. First, the sentence prior to a think aloud prompt likely serves as a retrieval cue for knowledge that supports mental model construction, which constrains the contents of working memory that are reported at the prompt. Presumably, these contents reflect the products of the processes that support mental model construction. Second, the instructions typically used with this approach may invoke a strategy to focus on thoughts that are reflective of mental model construction.
Approaches to Coding Verbal Protocols. Verbal protocols can be used to answer research questions regarding strategic processing to the extent that a coding system is developed that is aligned with those research questions. A coding system can be developed through an inductive process that arises from qualitative methodologies, or it can be developed a priori, grounded in theory. In this section, we discuss a few notable examples of these different approaches.
Pressley and Afflerbach (1995) reviewed an extensive list of studies that employed an open-ended prompt methodology (i.e., readers chose when to think aloud) for collecting verbal protocols and a traditional, qualitative analysis. These studies suggested that skilled readers regularly monitor cognitive states as they read, and engage in a number of strategies in response to challenges they face in terms of learning from and comprehending texts (e.g., rereading, paraphrasing, elaborative inferencing).
Trabasso and Magliano (1996) provided an example of developing a coding system a priori inspired by theory. They had college students think aloud after each sentence of short narratives. The coding system was based on the constructionist theory of comprehension (Graesser, Singer, & Trabasso, 1994) and distinguished between three broad categories of inferences: explanations, predictions, and associations. Consistent with the theory, Trabasso and Magliano (1996) found that explanations predominated when examining the inferences produced while thinking aloud. Moreover, explanations based on prior text information provided the primary basis for establishing how distal sentences were related, supporting global coherence. The results were thus consistent with the assumption that comprehension is achieved largely through explanatory reasoning (e.g., Graesser et al., 1994).
A number of researchers have developed coding systems sensitive to such theoretically based distinctions, including a broader range of strategies that can be reflected in verbal protocols (Kaakinen & Hydna, 2005; Magliano & Millis, 2003; Magliano, Millis, the RS AT Development Team, Levinstein, & Boonthum, 2011; McCarthy & Goldman, 2019; McMaster et al., 2012; Rapp, van den Broek, McMaster, Kendeou, & Espin, 2007; Todaro et al., 2008). These strategies concern paraphrases, episodic recollections, bridging inferences, elaborative inferences, metacognitive judgments, evaluative statements, and affective judgments. Theoretically derived strategies that directly support the construction of a mental model (e.g., bridging inferences, elaborations) tend to be more frequent in verbal protocols than are other strategies (Rapp et al., 2007; Todaro et al., 2008).
However, the frequencies of different strategies vary with reading proficiency (Mag-liano & Millis, 2003; McMaster et al., 2012). For example, Magliano and Millis (2003) had college students read simple narrative texts and think aloud at selected sentences that afforded bridging inferences based on an a priori, theoretical analysis of the causal structure underlying the story events. They found that proficient readers tended to produce more bridging inferences than less proficient readers, whereas less proficient readers tended to produce more paraphrases than proficient readers. Rapp et al. (2007) and McMaster et al. (2012) compared different profiles of struggling middle school readers who thought aloud at every sentence while reading. While one profile group of struggling readers tended to paraphrase the sentence that was just read, another profile group primarily produced invalid elaborative inferences, both approaches indicating difficulties using deep-processing strategies.
Validating Verbal Protocols. Despite arguments that thinking aloud is minimally subject to task demands (Ericsson & Simon, 1993), it is possible that participants produce thoughts that would not occur during silent reading. Especially, when participants are asked to reconstruct their thoughts, the methodology is prone to fabrication. As such, it is crucial to adopt approaches that help identify which aspects of verbal protocols likely support comprehension and text-based learning in contexts where readers are not asked to think aloud. There are at least three approaches to validating verbal protocols.
The first approach is to demonstrate that the strategies resulting from verbal protocol analysis are correlated with measures of individual differences known to affect comprehension, such as proficiency in reading (McMaster et al., 2012) and comprehension (Kopatich, Magliano, Millis, Parker, & Ray, 2019; Magliano & Millis, 2003; Magliano et al., 2011; Millis, Magliano, & Todaro, 2006), working memory capacity (Whitney, Ritchie, & Clark, 1991), proficiency in a second language (Zwaan & Brown, 1996), or disciplinary expertise (Graves & Frederiksen, 1991). The studies of Magliano and Millis (2003) and McMaster et al. (2012) cited above exemplify this type of validation. Zwaan and Brown (1996) provided a notable example with respect to demonstrating differences in strategic processing as a function of proficiency in a second language. They recruited participants that varied in knowledge of French and had them think aloud (in their native English) while reading narratives written in French or English. While reading in English, both groups of students demonstrated results consistent with Trabasso and Magliano (1996), with the most dominant strategy being explanation. While reading in French, however, only the proficient French readers demonstrated an explanation-based strategy, whereas the less proficient readers engaged in strategies that involved identifying the meaning of lexical items and syntactic relationships. As a final example of this approach, low working memory span readers have been shown to engage in excessive elaboration when thinking aloud compared to high span readers (Whitney et al., 1991).
A second approach to validating verbal protocol analysis is to demonstrate that strategies revealed by thinking aloud are correlated with measures of comprehension outcomes (Goldman et al., 2012; Kopatich et al., 2019; Magliano & Millis, 2003; Magliano et al., 1999, 2011). For example, Magliano, Millis, and colleagues have shown that measures of bridging inferences are positively correlated with comprehension outcomes for texts used in collecting the verbal protocols (Magliano & Millis, 2003;
Magliano et al., 2011; Millis et al., 2006), as well as for other texts (Magliano & Millis, 2003; Magliano et al., 2011). Conversely, the measures of paraphrasing tend to be negatively correlated with comprehension performance (Magliano & Millis, 2003). Magliano and Millis (2003) argued that more proficient readers engage in strategies that promote coherence building, whereas less proficient readers engage in locally focused strategies directed at understanding individual sentences. Kopatich et al. (2019) found that measures of bridging and elaborative inferences partially mediated the effects of general and language processing resources on comprehension outcomes.
A final approach for validating verbal protocols is to demonstrate that they are correlated with moment-to-moment measures of reading, which would suggest that these processes occur during silent reading. The study by Magliano et al. (1999) is an example of this approach. In that study, the explanations, predictions, and associations included in the verbal protocols of one group of students predicted variations in reading times in another group of students who read the same texts silently under different goal conditions (e.g., reading to explain vs. reading to predict). As another example, Kaakinen and Hydna (2005) collected verbal protocols and eye movement data while participants read expository texts, examining the length of fixations at sentences where participants were asked to think aloud. Those authors found significant correlations between the verbal protocols and eye movement data, with questions and explanations in the protocols, in particular, being positively correlated with fixation times.