As is the case for contextualized SJTs, the development process of general domain knowledge SJTs can be categorized into three main steps. However, as compared to contextualized SJTs, each of these steps differs when applied to the measurement of general domain knowledge.
Step 1: Item stems According to the general domain knowledge perspective, each stem needs to be designed in such a way that the stem activates the constructs reflected in the response options, thereby allowing people to show their varying levels of procedural knowledge about these targeted constructs. This means that the test designer should adopt a strategy to develop item situations (item stems) on the basis of theoretical frameworks or taxonomies so that these situations can activate specific behaviour related to the targeted traits or compound traits (or competences; Motowidlo et al., 2006a; Patterson, Ferguson, Norfolk & Lane, 2005). In other words, under the domain-general design scheme, the development of item stems mainly follows a deductive approach rather than an inductive approach. However, to guarantee the job-relatedness of the situations, it is sometimes important (though not absolutely necessary) to ‘beef up’ these situations with information from critical incident interviews or workshops. In any case, test developers are advised to keep the situation descriptions quite generic. An SJT measuring general domain knowledge requires just enough job-specific contextualization to make the SJT face valid and job-related.
Step 2: Response options, response instructions, and response format Collecting response options for general domain knowledge SJTs does not require a group of experienced SMEs with considerable job-specific knowledge about the domains to be tapped by the SJT, because the response options are intended to tap expressions of general domain knowledge. For instance, a sample of novices or industrial and organizational psychology students (because they have knowledge of traits and trait expressions) can be instructed to generate response options by asking them what they think would be the best way to handle the situation presented in each item stem (Motowidlo et al., 2006a). The test developer then edits these responses. A sample of 5-10 response options are then typically retained per item stem, with an equal number of response options that express high levels of the trait and low levels of the trait (effective vs. ineffective options).
To assess general domain knowledge, a knowledge-based response instruction format (‘What should you do?’) seems to be most appropriate. Applicants should be instructed to give effectiveness ratings for each option. In that case, the response format is typically a Likert-type scale rating format, although pick best/worst and rank order formats are also possible.
Figure 11.2 Example of general domain knowledge SJT item (related to agreeableness). Source. Motowidlo, Hooper & Jackson (2006a).
Step 3: Scoring key SMEs with extensive knowledge about the varying personality trait expressions in the response options are required to develop the scoring key. For the measurement of the personality trait conscientiousness, for example, personality psychologists or doctoral students in the domain of personality psychology could be approached to rate the response options. To this end, Likert-type scales can be used with verbal labels indicating the level of the trait expressed by the response option (e.g., 1 = very introverted to 7 = very extraverted; see Motowidlo & Beier, 2010). Agreement levels should be computed by comparing the ratings across judges and by comparing the ratings with a priori trait levels that the response options were designed to express.
In Figure 11.2, we present an example of a general domain knowledge SJT item that was taken from Motowidlo and colleagues (2006a). Contrary to contextualized SJTs (see Figure 11.1), the description of the situation is more generic and more widely applicable across many job situations and is specifically intended to serve as a framework for the measurement of a particular construct (in this case the personality trait agreeableness). Another difference is that the response options were specifically written to measure agreeableness. Whether the response options are indicative of high or low levels of agreeableness is mentioned in parentheses. People who rate those options that express high levels of the personality trait positively and those options that express low levels of the personality trait negatively are believed to be in high possession of the trait and have general domain knowledge about how to express this trait effectively in work situations.