Assessment Frameworks and ‘Competence’
Once the concept of assessing ‘CRM’ skills was accepted, the best mode of sampling was considered to be observation on the flight deck and, to that end, various marker frameworks were proposed as tools for capturing performance in the workplace. There are three methods commonly available for developing assessment schedules. The first method looks at past failure and attempts to identify behaviours that would have reduced the probability of an undesired outcome. Probably the longest-standing
Extract from University of Texas Behavioural Markers Rating Scale
aviation marker framework, the NASA-UT Crew Effectiveness Markers, represents this approach. It formed the basis of the subsequent University of Texas Behavioural Markers Rating Scale. Table 11.1 illustrates two of the 13 suggested markers, each of which was rated on a four-point scale (Helmreich et al., 1990)
The next approach is the ‘expert committee’ model. This method involves panels of subject matter experts debating what acceptable behaviour looks like. The European NOTECHS scheme (Flin et al., 2003) and the more recent EBT/CBT framework (EASA, 2019) exemplify this approach and is, possibly, the most common method used. Examples are at Tables 11.2 and 11.3.
The third approach is to conduct structured interviews with operating crew, using tools such as critical incident and repertory grid, to seek examples of behaviour both good and bad. Having assembled a range of statements, subject matter experts then undertake a ‘card sort’ activity’ (MacLeod, 2005). The set of statements are assigned to categories according to similarity or relatedness. An example is in Table 11.4.
In previous chapters I have used data from a number of line operations safety audit (LOSA) surveys. A content analysis of the narrative reports written by observers is another source of information about crew performance. Table 11.5 illustrates a structure that emerged from 423 comments extracted from reports. Only the most significant clusters are included.
Categories and Elements of NOTECHS
Extract from EASA Competency Framework
Interview-based Marker Construction
Emergent Structure from LOSA Narratives (% of comments)
The approaches illustrated differ in that the first three - the UT, NOTECHs and EBT frameworks - reflect the views of those involved in their construction. They are heavily influenced by the opinions of the collaborators and this is sometimes apparent in the language used in the descriptions. The latter two are rooted in the observation of workplace performance and the structure emerges later. One fundamental weakness of these approaches is that they were applied to the problem of developing a process of performance assessment, not to elucidate a comprehensive competence model. They were designed to sample, not to catalogue. By default, then, they will be incomplete in that their coverage of performance wall not be comprehensive. Also, their formulation wall reflect their intended purpose.
A further problem is that the frameworks described were designed to be used in any context. No consideration was given to whether different categories of aviation demand different competencies. For example, as we have seen, business aviation pilots have to spend time managing the expectations of their clients as well as flying the aircraft. Freighter crews have to deal with complex operational issues, often to do with the cargo they are carrying, while at remote destinations with little operational support. Police and emergency medical support helicopters often have to make decisions under intense pressure. While it is probable that there is a core of common performance across all types of aviation, we also need to address the complete operational envelope, to match training to the risk profile and to the operational demands faced by the user. In the next section, I want to explore what competence requirements emerge from looking at aviation from a systems perspective.