Addressing concerns about self-report measures

Self-report measures, such as questionnaires and surveys, are often easily implemented to gather a large dataset. Another reason for the preference for self-reports is that they provide a time- and cost-effective way to collect data (DeBacker, Crow-son, Beesley, Thoma, & Hestevold, 2008). Therefore, it is not surprising that selfreporting has been a dominant data collection regime in the field of research on individuals’ conceptions of knowledge. Although self-report methods are widely used, many researchers nevertheless criticise this method. Among other things, researchers have found that individuals’ conceptions of the nature of knowledge and knowing are extremely difficult to measure with self-report assessments (DeBacker et al., 2008; Karabenick et al., 2007; Muis, Duffy, Trevors, Kanellucci, & Foy, 2014; Muis et al., 2016; Schraw, 2013).

One challenge in investigating an individual’s conception of knowledge and knowing is that this kind of phenomenon is not directly observable. The interpretations of an individual’s conception of knowledge are thus for the most part indirect. Furthermore, epistemic conceptions are characteristically abstract constructs. Therefore, reporting these kinds of conceptions requires considerable self-reflection and abstraction from respondents and it is cognitively, extremely challenging (Karabenick et al., 2007). The most important drawback to self-report assessments is that individuals are to some extent unable to introspectively assess themselves (Bowman, 2010). Previous research has also shown that students’ belief in themselves as knowers is not necessarily equivalent to how they perform and assess knowledge in real life (e.g., Hyyti-nen, Holma, Toom, Shavelson, & Lindblom-Ylanne, 2014). Moreover, individuals are not necessarily aware of their own conceptions and thus they are not competent to describe their perceptions (Bowman, 2010). Therefore, selfreport measures may provide incorrect information despite the respondents’ best efforts to be honest and accurate in the data collection situation. It has been suggested that self-reports alone cannot adequately assess complex phenomena (Greene & Yu, 2014).

In addition, stronger critiques have questioned the reliability and validity of present self-report assessments (Bowman, 2010; DeBacker et al., 2008; Karabenick et al., 2007; Muis et al., 2014). Previous studies have identified several problematic issues with questionnaires assessing individuals’ conceptions of knowledge (e.g., Buehl & Alexander, 2005; DeBacker et al., 2008; Hyytinen et al., 2016; Muis, Bendixen, & Haerle, 2006; Muis et al., 2014). One concern relates to tests and how the phenomenon is operationalised and conceptualised. Students’ interpretations of present self-report items have been found to be inconsistent with researchers’ assumptions and intended meanings (Greene, Tomey-Purta, & Azevedo, 2010; Muis et al., 2014). Previous research has shown that there is clear variation not only between but also within student groups in how students understood, interpreted, and responded to items concerning epistemic conceptions (Muis et al., 2014).

Several reasons for these conceptual shortcomings can be given. Firstly, one reason for inconstancies is that self-report items often include complex concepts (i.e., “truth”, “expert”, “first-hand knowledge”), which need to be interpreted and combined with relevant contexts and experiences when responding (see Karabenick et al., 2007; Muis et al., 2014). Secondly, some items have been found to include multiple or ambiguous meanings and interpretations (Greene et al., 2010; Hyytinen et al., 2016; Muis et al., 2014) as the following extract from a focus group interview of Finnish students in educational sciences shows:

Hofer’s DEBQ questionnaire item “Truth is unchanging in this subject”

(see Hofer, 2000, p. 390)

Student 8: For me truth is a matter that is considered correct.

Student 7: Yeah, research tells us what is generally accepted in a specific moment within a particular context.

Student 6: But I considered here that [the meaning of “truth” | was a philosophical view, not a verified fact or something.

Student 1: I thought this [truth] referred to the construction of knowledge.

Student 6: So. What is actually meant [by this item]? This question is really paralysing! For me nothing holds absolute truth but some aspects can be accepted as truthful at a particular moment in a specific context.

Hyytinen and colleagues (2016) found that similar challenges with the questionnaire items in the DEBQ resulted in a high number of missing values. In addition, the problematic items included several “unsure” responses (response alternative 3 on a 5-point Likert scale). Furthermore, related factor analysis resulted in an unclear factor solution, including low commonalities with some items. In a similar vein, DeBacker and colleagues (2008) analysed data from three self-report questionnaires, namely the Epistemological Questionnaire (EQ) (Schommer, 1990), the Epistemic Beliefs Inventory (EBI) (Schraw, Bendixen, & Dunkle, 2002), and the Epistemological Beliefs Survey (EBS) (Wood & Kardash, 2002). Their results indicated psychometric problems with all three. The results demonstrated, among other things, consistent failure of factor analyses (exploratory and confirmatory) to support the hypothesised factor structures. In addition, the reliability of the scales remained rather low, and the scales functioned differently in different contexts. The use of a Likert scale in such measurements has also been debated elsewhere (e.g., Greene et al., 2010; Muis et al., 2006, 2014). Muis and her colleagues (2014) found that students chose option 3 when conflicting items occurred. In contrast, Greene and colleagues (2010) reported that students opt for “3” as a neutral response when they were unfamiliar with the question.