Ways to improve the validity of references
In light of the literature, it is clear that the extent to which employers use and rely on references is unjustified and not supported by research evidence. However, research in this area provides some useful guidelines to improve the validity of recommendation letters.
First, it has long been suggested that forced-choice items (e.g. ‘Does X like working in a team or working alone?’) reduce the effects of overall leniency and can increase accuracy (Carroll & Nash, 1972). Yet forced-choice items must be carefully selected, and even then candidates could be described by either extreme as items are rarely mutually exclusive.
Second, employers should count key words (e.g., able, creative, reliable), previously determined on the basis of job analysis. This technique provides some order to unstructured references, though it is certainly not immune to the referee’s style. Peres and Garcia (1962) scrutinized over 600 references and identified five key areas that can be used to organize the key word count: cooperation, intelligence, extraversion (‘urbanity’), vigour and conscientiousness (‘dependability’). Three decades later Aamodt, Bryan and Whitcomb (1993) analysed students’ references and found support for these categories. Although it is questionable whether these categories truly represent the best way to organize and classify the content of references - notably because established personality taxonomies, such as the Big Five, and cognitive ability models (see chapters 7 and 6, respectively) have a stronger and more generalizable theoretical basis - it is clear that having a taxonomy or framework to assess unstructured references does help.
Third, the predictive validity of references tends to increase when referees are asked to use relative percentiles (comparative rankings of how well the candidate does in any given area relative to the group the referee uses as a frame of reference). Although percentiles are not normally distributed and inflated (80th percentiles being the average; McCarthy & Goffin, 2001), they still force referees to distinguish between candidates.
Fourth, it has been argued that if the anonymity of the referees were preserved, references would be less lenient, more varied and more accurate and valid (Ceci & Peters, 1984).
Research also indicates that using concrete examples to back up statements about the candidate’s attributes and including both positive and negative information about the candidate leads to better references. This was Knouse’s (1983) conclusion. The worst-case scenario, on the other hand, was for references that had no examples and included some negative information.