Desktop version

Home arrow Philosophy

  • Increase font
  • Decrease font

<<   CONTENTS   >>

Acknowledging the inferential gaps between what is heeded in mind and what indeed is verbalized

As much as we value the scientific merits of verbal protocol analysis, we also acknowledge several limitations that need to be addressed with care. Pressley and Afflerbach (1995) portrayed verbal protocol analysis as a “maturing” methodology (p. 1). We agree with their point that, while much interesting work has been already accomplished with varied verbal protocol analysis approaches, the ongoing development of the methodology parallels the evolution in our understanding of acts of reading. A result can be continued refinement of both.

Perhaps the most difficult task is to identify and narrow the inferential gap that exists between what we desire to understand (describe, reveal, or verify) and what we can do with data (Messick, 1989). Verbal protocol analysis is not free of this concern because what is inferred from verbal data might neither fully nor accurately represent thoughts and processes of readers (Ericsson & Simon, 1998; Smagorinsky, 1998). For example, it is obvious that what we desire to know is a “pure” form of situated processes (i.e., cognitive strategies) and thoughts (i.e., mental representations by the strategy use) which are heeded in mind (i.e., short-term memory) during an authentic cognitive task. These processes and thoughts—namely, first-level cognition “Cl” in this discussion—could be those that are not intervened by any additional construct-irrelevant tasks imposed by others. However, psychological realities of Cl processes and thoughts may not be explicitly observed by any research technologies currently available. To address the issue, alternative ways of collecting data which are assumed to best characterize the genuine Cl processes and thoughts are necessitated.

However, this situation (i.e., with the goal of better understanding authentic cognition by adding a somewhat unnatural task or mechanism in order to observe that cognition) may cause researchers to be in a dilemma where the additional data-gathering task, especially when involving extra language generation, may facilitate modifications of strategic processing and thinking at the very moments of data generation—namely, second-level cognition “C2” here. That is, verbal reporting inevitably influences the operation of Cl-level processing and thinking and what it captures may be those at the C2-level because language mediates cognitive work (e.g., Davis, Carey, Foxman, & Tarr, 1968; Posner, 1982; Smagorinsky, 1998). Such influences are oftentimes indicated by readers slowing down the reading speed or being disoriented in repeating surface-level comments on thoughts. Also, there is a chance that some of the chains of strategic thinking and related action are non-verbalized for a particular reason (e.g., personality, motivation, verbal proficiency, working memory at capacity, relational factors) and are consequently excluded from the analysis. What matters at this point is with what rigor and in relation to what context the investigator can account for the involvement of task demands, as well as the unexpectedly modified acts of reading, in describing the C2 thoughts and processes to the greatest extent.

In a worst scenario, verbal reports may be an inaccurate and unreliable source of data when participants’ verbalizations significantly change the course of thinking and processing. Such uncontrolled interruptions may, in turn, entirely disrupt what readers normally do and think in a task that would otherwise be authentic and ecologically valid. What is verbalized in this case are of third-level cognition “C3,” which might not be in our best interests because what we are striving for is to estimate and fill in the data-reality gap while maintaining the substance and sequence of strategic processing and thinking that best resemble those for authentic reading.

Nonetheless, we believe that the major events of strategic text processing, as well as the mental representations that emerge across the moments of processing, can be generally accessible and reportable in a well-designed research task (Fox, Ericsson, & Best, 2011). Figure 23.1 illustrates that it is highly possible that, despite the influences, to some extent, the task of verbal reporting would not significantly change the course of reading strategy use and the properties of represented thoughts (Norris, 1990). It means that the inferences we propose to make from verbal reports should represent the course of cognitive actions, metacognitive controls, and evolving representations at the C2 level without much losing the integrity of Cl-level authentic reading.

It is important to note that the integrity of verbal protocols is the result of text- and task- appropriate design. Such design accounts for all relevant experimental variables (e.g., prompts, waiting time, text familiarity, task impressions) as well as the contextual factors (e.g., reading goals, materials and resources, settings and situations, textual scopes and boundaries). A carefully designed verbal-reporting task minimizes the likelihood of over-involvement of automatic processing (e.g., if a reading task is familiar and easy enough for the reader to complete it quickly without much effort) or unnecessary facilitation of vague self-explanations (e.g., the task is too difficult to complete so the reader becomes constantly relying on the help of self-talk without a clear focus and deep engagement in text understanding). When conducting data analysis,

A Flow of Thinking in Verbal Reporting during Reading

Figure 23.1 A Flow of Thinking in Verbal Reporting during Reading

the investigator should carefully examine the data, progressing into a better situated understanding of the verbalized thoughts while self-questioning whether data makes sense in relation to a particular coordination of reader, text, and context. Depending on the focus of the particular study, researchers may also conduct grounded theorizing (Glaser & Strauss, 1967) in which new insights into reader-text-task-context interactions can be generated from verbal report data.

Opportunity is that acknowledging limitations of a method may fuel the action to design a rigorous research task and data analysis. For verbal protocol analysis, the inferential gap in the verbal data analysis would make it difficult to evaluate the proximity and disparity between what is heeded in mind and what indeed is verbalized. However, considering taken-for-granted challenges seriously may create a new opportunity to generate, test, and apply alternative principles and means to address manifold issues in verbal protocol analysis. When researchers challenge commonplace assumptions, develop alternative strategies, and explore new approaches, informed research processes may contribute to constructing a contextualized account of strategic processing as close as possible to that which is carried out in an authentic task of reading.

In what follows, we focus on the specific merits and major issues in verbal protocol analysis and possible means of addressing the issues, as we seek to undertake rigorous and situated investigations of the strategic processing involved in complex reading tasks. We draw on our own previous work, and that of others, to discuss theoretical and practical considerations for qualitative analysis of verbal report data. Specifically, we focus on:

  • • how the contextual knowledge of the investigator creates new opportunities for verbal protocol analysis,
  • • how data saturation contributes to understanding the phenomena being interpreted and is judged in ways that inform the progress of qualitative analysis, and
  • • how evidentiary reasoning is undertaken rigorously throughout the analysis and contributes to the unique conceptual contributions that a qualitative approach holds.
<<   CONTENTS   >>

Related topics