Desktop version

Home arrow Philosophy

  • Increase font
  • Decrease font

<<   CONTENTS   >>

Eye Movement Cued Self-reports

One novel approach to eliciting self-reports of strategic processing is to record participants’ eye movements while they are performing a task, and afterwards display those eye movements to them to cue reports of their thinking during task performance. Eye movement cued self-reports can be considered a form of retrospective thinking aloud because participants are responding to external cues rather than internal cognitive states. Thus, they are retrospective in the sense that participants are asked to reflect upon strategic processing in the near past, with the products of those strategies likely still in working memory. To elicit these reports, participants are presented with a video of what they just saw while performing a specific task, with the video superimposing participants’ eye movements as a small circle indicating where they were looking during task performance. As eye movements are considered to reflect cognitive processes (Rayner, 2009), the rationale is that they may cue participants to report orally on such processes as they were taking place in a specific task environment. In particular, eye movements are assumed to represent an externalization of strategic processes that may be used to cue descriptions of those processes.

Eye movement cued self-reports have been used to measure strategic processing in a variety of learning tasks, including multimedia learning (de Koning, Tabbers, Rikers, & Paas, 2010; Stark, Briinken, & Park, 2018), text reading (Catrysse, Gijbels, & Donche, 2018; Penttinen, Anto, & Mikkila-Erdmann, 2013; Salmeron, Naumann, Garcia, & Fajardo, 2017), web search (Brand-Gruwel, Kammerer, van Meeuwen, & van Gog, 2017; Muntinga & Taylor, 2018), visual tasks (Jarodzka, Scheiter, Gerjets, & van Gog, 2010), mathematical problem solving (Rau, Aleven, & Rummel, 2017; van der Wei-jden, Kamphorst, Willemsen, Kroesbergen, & van Hoogmoed, 2018), and interaction with and inspection of complex systems (Ruckpaul, Fiirstenhbfer, & Matthiesen, 2015; van Gog, Paas, van Merrienboer, & Witte, 2005; van Meeuwen, Brand-Gruwel, Kirschner, de Bock, & van Merrienboer, 2018).

Issues Concerning the Presentation of Eye Movements. A critical aspect of eye movement videos concerns speed. As eye movements tend to be quick, participants easily may lose track of the fixations. Therefore, Russo (1979) recommended that the replay proceed at a rate compatible with recall and reports. Accordingly, several researchers have used slower speed for video replays, such as 50% (Brand-Gruwel et al., 2017; de Koning et al., 2010; Hyrskykari, Ovaska, Majaranta, Raiha, & Lehtinen, 2008) or 75% of the actual speed (van Meeuwen et al., 2018). Others have replayed the videos at full speed, however (Jarodzka et al., 2010; Salmeron et al., 2017; Stark et al., 2018). While it is evident that eye-movements are faster than verbalizations, it is not so obvious that each fixation represents a unique processing episode that deserves to be commented on. Accordingly, Guan, Lee, Cuddihy, and Ramey (2006) reported that 47% of eye movement visits to particular regions of interest did not elicit any verbal comments. This is especially clear when performing reading tasks, where participants may engage for minutes in reading a long paragraph. Slowing down the replay of this kind of episode may not provide new insights into participants’ ongoing processing because they are tapping the same process; rather, it may induce negative reactions toward the task. Researchers also have varied the way videos are replayed as a way to accommodate this issue. Thus, participants have been allowed to change the speed of the recordings (Russo, 1979) or to pause the videos at any time (de Koning et al., 2010; van Gog et al., 2005). Such procedures allow students to report complex thoughts in tasks where a sequence of fixations reflects different cognitive processes. In other cases, such as when a long sequence of fixations reflects a single process (e.g., deep reading), the researcher can just pause the video at points of critical interest (Penttinen et al., 2013; Ruckpaul et al., 2015).

Another relevant issue concerns how eye-movements are represented in the videos. While the major components of eye-movements include both fixations and saccadic movements, the majority of studies have presented eye-movements only as individual fixations, with the size of the circles representing the length of the fixations. During task performance, previous fixations disappear and new ones appear in different locations. This way of representing eye movements thus emphasizes location (i.e., fixation) over orientation (i.e., saccadic movement). It is therefore possible that it increases the likelihood that processing related to objects (e.g., describing objects) are elicited, compared to more dynamic processing (e.g., comparing and contrasting objects). In a rare exception, Ruckpaul et al. (2015) replayed videos representing eye movements during task performance as fixations (i.e., circles) as well as saccadic movements (i.e., lines between circles). Unfortunately, these authors did not include a condition showing only fixations, which made it impossible to address the possibility mentioned above.

Validating Eye Movement Cued Self-reports. Regarding validity, eye movement cued self-reports have been proposed as a less intrusive alternative to concurrent thinking aloud (van Gog et al., 2005). However, as noted earlier, given that participants are asked to reconstruct their thoughts, the methodology is prone to fabrications. Such fabrication may still be less frequent than for retrospective self-reports that do not use eye-movement cues because the reconstruction, at least, must be consistent with the sequence of fixations (Guan et al., 2006).

In the fields of usability and learning research, there have been some attempts to analyze the reliability and validity of this methodology by comparing it to concurrent thinking aloud. Not surprisingly, thinking aloud in response to cues provided by eye movements tends to involve a higher number of verbalizations (Brand-Gruwel et al., 2017; Hansen, 1991; Hyrskykari et al., 2008; Russo, 1979). This may be due, in part, to the fact that participants spend more time reporting when cued by eye movements. Once reporting time is controlled for, however, this difference seems to disappear (Ruckpaul et al., 2015). More importantly, the two methodologies may elicit different types of thoughts (Brand-Gruwel et al., 2017; Hansen, 1991; Hyrskykari et al., 2008; Ruckpaul et al., 2015; van Gog et al., 2005). In usability studies, where participants are assessed as they interact with a system (e.g., when buying products from different web pages), the percentage of manipulative comments (e.g., “I write the name into this field”) is lower with eye movement cued than with concurrent thinking aloud, while the difference in terms of cognitive comments (e.g., reflections about the system) is less clear (Hansen, 1991; Hyrskykari et al., 2008). In studies of learning, where participants are engaged in a task to meet a learning goal, the major difference between the two methodologies seems to be the number of metacog-nitive and evaluative comments produced (i.e., evaluations of the information and the learning process), with higher frequencies observed in eye movement cued verbal protocols (Brand-Gruwel et al., 2017; van Gog et al., 2005). In brief, such comparative studies suggest that protocols from concurrent thinking aloud reflect more processing concerning what participants are doing, while protocols from eye movement cued thinking aloud reflect more processing involving evaluation of their actions.

As mentioned earlier, a major threat for eye movement cued thinking aloud is fabrication. Guan et al. (2006) compared the sequences of verbalizations in such protocols to the actual sequences of problem solving as indicated by participants’ eye movements. More than 80% of participants’ verbalizations of what they were attending to corresponded to the actual sequences of their eye movements, and less than 3% of the verbalizations corresponded to areas of the task scenario that were not identified by the eye movement data. This pattern of results suggests a low rate of fabrication, but this point should be taken with caution. To the best of our knowledge, no other study has empirically tested the issue of fabrication with eye movement cued self-report data.

Thus far, there also have been few attempts to validate this methodology by testing its predictive power. Eiling, Lentz, and de Jong (2011) found that participants thinking aloud concurrently or retrospectively in response to eye movements did not differ in terms of the usability problems detected. Brand-Gruwel et al. (2017) reported a global positive correlation between evaluative verbalizations and performance (i.e., selection of appropriate web pages), but correlations were not reported separately for each methodology (i.e., concurrent and retrospective eye movement cued thinking aloud). Thus, any claim about the validity of this methodology should be considered with caution until further documentation is available.

<<   CONTENTS   >>

Related topics