Generalizing validity evidence
In addition to discussing the three types of validity described above, the Guidelines address the potential for a validity transportability procedure (Biddle & Nooren, 2006). In this process, evidence of validity in one employment context is ‘imported’ from another employment context. A caveat here is that the Guidelines describe transportability as utilizing the evidence of the same test in different contexts (Hanges et al., 2013). Several requirements must be met for transportability to be supported as appropriate. These requirements include: 1) job similarity in the two work contexts in relation to work behaviours, as demonstrated by a job analysis; 2) validity evidence for the test should meet the requirements outlined in the Guidelines; and 3) the test must demonstrate fairness for protected groups (Equal Employment Opportunity Commission, Civil Service Commission, Department of Labor & Department of Justice, 1978).
While the Guidelines speak directly to transportability, the Standards and Principles describe the broader concept of validity generalization. Validity generalization studies directly examine the robustness of validity inferences made from other contexts, including environments, jobs and even tests (Hanges et al., 2013). This process can be done through multiple techniques, including the use of meta-analysis and synthetic validity. A metaanalysis is a process in which the results of multiple studies are aggregated to examine the overall ability of a selection test to predict performance across contexts. As such, when a test has high average validity across contexts, there is no need to demonstrate the validity of the test in a specific context. Even when a meta-analysis supports the validity of a given test across contexts, there should still be evidence of the similarity in job requirements at the local level, as well as evidence of similarity in job requirements of those jobs involved in the meta-analysis (Schmitt & Sinha, 2011). Further complicating the issue of validity generalization, case law to date seems to both support and refute the use of validity generalization using meta-analysis. For that reason, Landy (2003) suggests supplementing the use of meta-analyses with transportability analyses and ensuring that a proper and detailed job analysis is conducted.
Similarly, synthetic validity (also referred to as job component validity) is a process for inferring the validity of a selection test or set of tests in a particular situation based on the elements of a job (Johnson et al., 2010). This process is most often used when an employer does not have enough incumbents to conduct a criterion validity study for a particular job or set of jobs. In this case, a group of jobs is analysed based on the key components of those jobs, including their tasks and work behaviours as determined by a job analysis. Tests are then selected that are expected to predict the key components of those jobs. Finally, the relationship between these job components and the predictor tests is estimated (Scherbaum, 2005). A test battery can then be built using the validity inferences of the job components and predictors.
Synthetic validity overcomes the issue of small incumbent sample sizes by aggregating the same job components across different jobs. While synthetic validity has many benefits, including providing stable validity estimates and enabling mass production of tests, there are still legal risks (Johnson et al., 2010). So far just two court cases have addressed synthetic validity and neither gave a direct endorsement of the procedure. However, the legal risks associated with synthetic validation are inherent in most types of validity generalization. Therefore, employers must examine the limitations they have for estimating validity coefficients and determine whether a validity generalization approach that uses best practices, as outlined in the Principles and Standards, is the best route for them.
After demonstrating the job-relatedness of a selection process through validation procedures, and convincing the judge that the selection process is indeed job-related, the burden of proof shifts back to the plaintiff(s). At this point it is their responsibility to demonstrate that the employment practice is a pretext for discrimination. This can be done by providing evidence of other selection processes that are equally valid and result in less adverse impact (Hanges et al., 2013).