Without a robust theory, establishing the measurement properties of PROMs is often a viciously circular endeavor. Consider construct validity. Construct validity asks whether the questionnaire at hand performs in a way that the underlying theory suggests it should. An increasingly common criticism of construct validation in the context of classical test theory—the dominant psychometric paradigm—is its inability to determine if a measure represents its object of inquiry, i.e., its inability to provide evidence of validation (Hobart et al. 2007).
Construct validity is typically tested by assessing a measure’s internal and external validity. Internal construct validity is tested by examining the extent to which the questions or items within a measurement scale are statistically related to one another based on the responses given by a sample population. But this process does not tell us anything about the construct itself, e.g., quality of life or mobility. It tells us only that certain questions tend to occupy the same conceptual space. External construct validity is examined via convergent and divergent validity testing. Here, multiple measurement scales deemed similar to and different from one another are applied to a sample population, and the scores derived from respondent answers are correlated. These correlations determine whether the scale being validated correlates higher with scales that measure similar constructs than with those measuring dissimilar constructs. Once again, this process does not tell us what construct a measure actually assesses. It tells us only that some scales are correlated (or not) with other scales. But without some kind of theory, it is not clear what any of these measures actually measure.