Predictive validity
View on Wikipediafrom Wikipedia
In psychometrics, predictive validity is the extent to which a score on a scale or test predicts scores on some criterion measure.[1][2]
For example, the validity of a cognitive test for job performance is the correlation between test scores and, for example, supervisor performance ratings. Such a cognitive test would have predictive validity if the observed correlation were statistically significant.
Predictive validity shares similarities with concurrent validity in that both are generally measured as correlations between a test and some criterion measure. In a study of concurrent validity the test is administered at the same time as the criterion is collected. This is a common method of developing validity evidence for employment tests: A test is administered to incumbent employees, then a rating of those employees' job performance is, or has already been, obtained independently of the test (often, as noted above, in the form of a supervisor rating). Note the possibility for restriction of range both in test scores and performance scores: The incumbent employees are likely to be a more homogeneous and higher performing group than the applicant pool at large.
In a strict study of predictive validity, the test scores are collected first. Then, at some later time the criterion measure is collected. Thus, for predictive validity, the employment test example is slightly different: Tests are administered, perhaps to job applicants, and then after those individuals work in the job for a year, their test scores are correlated with their first year job performance scores. Another relevant example is SAT scores: These are validated by collecting the scores during the examinee's senior year and high school and then waiting a year (or more) to correlate the scores with their first year college grade point average. Thus predictive validity provides somewhat more useful data about test validity because it has greater fidelity to the real situation in which the test will be used. After all, most tests are administered to find out something about future behavior.
As with many aspects of social science, the magnitude of the correlations obtained from predictive validity studies is usually not high.[3] A typical predictive validity for an employment test might obtain a correlation in the neighborhood of r = .35. Higher values are occasionally seen and lower values are very common. Nonetheless, the utility (that is the benefit obtained by making decisions using the test) provided by a test with a correlation of .35 can be quite substantial. More information, and an explanation of the relationship between variance and predictive validity, can be found here.[4]
Predictive validity in modern validity theory
[edit]The latest Standards for Educational and Psychological Testing[5] reflect Samuel Messick's model of validity[6] and do not use the term "predictive validity." Rather, the Standards describe validity-supporting "Evidence Based on Relationships [between the test scores and] Other Variables."
Predictive validity involves testing a group of subjects for a certain construct, and then comparing them with results obtained at some point in the future.
References
[edit]- ^ Cronbach, L.J., & Meehl, P.E. (1955). Construct validity for psychological tests. Psychological Bulletin, 52, 281–302.[1]
- ^ The Marketing Accountability Standards Board (MASB) endorses this definition as part of its ongoing Common Language in Marketing Project.
- ^ "Where Predictive Validity May Fail To Make The Grade".
- ^ "Do Psychometric Tests Work?".
- ^ American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (1999). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.
- ^ Messick, S. (1995). Validity of psychological assessment: Validation of inferences from persons’ responses and performances as scientific inquiry into score meaning. American Psychologist, 50, 741–749.
Predictive validity
View on Grokipediafrom Grokipedia
Definition and Fundamentals
Core Concept
Predictive validity refers to the extent to which scores on a test or measure can accurately forecast future performance or outcomes on a relevant criterion.[1] This form of validity evidence is gathered through empirical relationships between the test scores and subsequent criterion measures, demonstrating the test's utility in anticipating real-world behaviors or achievements.[1] At its core, predictive validity involves two primary components: the predictor variable, typically the score from a test or assessment, and the criterion variable, which represents the future outcome or behavior the predictor aims to estimate, such as job performance or academic success.[1] The predictor is administered first, followed by observation of the criterion after a meaningful time interval, ensuring the assessment's relevance to longitudinal forecasting rather than contemporaneous evaluation.[1] This temporal separation distinguishes predictive validity from concurrent validity, which examines relationships between measures taken at the same time.[1] A classic illustration of predictive validity is the use of college entrance exams, such as the SAT, to predict first-year grade point average (GPA). Recent meta-analyses have shown that SAT scores correlate moderately with subsequent college GPA, typically accounting for around 10-20% of the variance in academic performance one year later, highlighting the exam's role in identifying students likely to succeed in higher education.[3] Recent studies, including those on the digital SAT introduced in 2024, continue to demonstrate similar predictive power, though high school GPA often explains additional variance when combined with test scores.Distinction from Other Forms of Validity
Predictive validity, as a subtype of criterion-related validity, is distinguished by its emphasis on the temporal separation between test administration and criterion measurement, focusing on the test's ability to forecast future outcomes. In contrast, concurrent validity evaluates the correlation between test scores and a criterion measured at approximately the same time, often used to establish immediate alignment with established measures without the need for longitudinal follow-up. This distinction arises from the foundational classifications in early psychometric standards, where predictive validity requires evidence of future performance prediction, such as job success from aptitude tests, while concurrent validity supports current status assessments, like correlating a new depression scale with an existing one administered simultaneously. Unlike content validity, which assesses whether a test adequately samples the relevant domain or knowledge area through expert judgment and item representativeness, predictive validity prioritizes empirical correlations with external outcomes rather than the test's structural fidelity to the construct's universe. Content validity ensures the test covers all necessary aspects of the measured domain, such as ensuring an arithmetic test includes diverse problem types, but it does not inherently address predictive power. Predictive validity, therefore, complements content validation by demonstrating practical utility in anticipating real-world behaviors or achievements, without relying on subjective domain coverage evaluations. Predictive validity also differs from construct validity, which broadly examines the theoretical alignment of a test with an underlying psychological construct through multifaceted evidence, including convergent and discriminant correlations. While predictive validity focuses narrowly on criterion prediction as one form of construct validation—such as linking intelligence test scores to later academic success—construct validity encompasses a wider array of inquiries, like nomological networks and theoretical consistency, to confirm the test measures the intended abstract trait. This subtype relationship highlights how predictive evidence contributes to but does not exhaust construct validation. In modern psychometric frameworks, predictive validity fits within the broader category of criterion-related validity, which itself nests under "relations to other variables" as one of five sources of validity evidence, alongside test content, response processes, internal structure, and consequences. This hierarchical integration, shifting from rigid types to accumulative evidence, underscores that predictive validity provides specific empirical support for score interpretations but must be evaluated alongside other sources for comprehensive validity arguments. Overlaps occur when predictive studies inform construct or content evidence, but the core focus remains on future-oriented criterion correlations.[4]Historical Development
Origins in Psychometrics
The concept of predictive validity emerged in the late 19th century as part of the broader psychometric movement focused on quantifying individual differences for forecasting outcomes. Francis Galton, a pioneer in anthropometrics, established early foundations through his work in the 1880s and 1890s, where he measured physical and sensory traits to identify variations among individuals and explore their implications for inheritance and ability prediction.[5] His anthropometric laboratory at the International Health Exhibition in 1884 collected data on thousands of participants, emphasizing statistical methods to predict traits from measurable differences, thus laying groundwork for later predictive applications in psychology.[6] In the early 20th century, E.L. Thorndike advanced these ideas through correlational approaches in educational psychology. In his 1904 book An Introduction to the Theory of Mental and Social Measurements, Thorndike introduced methods for using correlations to predict educational outcomes from test scores, promoting the idea that mental measurements could forecast future performance by linking current assessments to later achievements.[7] This work emphasized empirical prediction via statistical associations, influencing the shift toward using tests for practical forecasting in schools before the 1930s.[8] A key milestone came during World War I, when the U.S. Army's Alpha and Beta tests (developed in 1917–1918 under Robert Yerkes) represented the first large-scale application of predictive validity. These intelligence tests were designed to predict soldiers' training success and job performance, correlating scores with outcomes like leadership potential and task efficiency across over 1.7 million examinees.[9] The effort demonstrated predictive utility in high-stakes selection, though correlations were modest (around 0.3–0.5 for performance prediction), highlighting early challenges in accuracy. Truman L. Kelley's 1927 publication Interpretation of Educational Measurements further formalized predictive elements within validity, defining it partly as the test's prognostic value—the correlation between scores and subsequent success in educational or occupational settings.[10] Kelley stressed that true validity required evidence of such future-oriented predictions, integrating correlational techniques from prior works to evaluate tests' practical utility.[11]Evolution Through Key Publications
In the late 1930s, Louis L. Thurstone advanced the understanding of predictive validity through his work on multiple-factor analysis, particularly in his 1938 publication Primary Mental Abilities, where he demonstrated how batteries of tests could predict performance by isolating distinct aptitude factors such as verbal comprehension and spatial visualization.[12] This approach emphasized the predictive power of multifaceted psychological measures over single-factor models, laying groundwork for more nuanced forecasting in psychometric applications. During World War II, predictive validity gained practical prominence in personnel testing, as military organizations employed aptitude tests to forecast soldier performance and suitability for roles, with studies showing correlations between test scores and on-the-job success that informed large-scale selection processes.[13][14] The 1950s marked a formalization of predictive validity within professional standards, highlighted by the American Psychological Association's (APA) 1954 Technical Recommendations for Psychological Tests and Diagnostic Techniques, which explicitly categorized predictive validity as a distinct type involving correlations between test scores and future criteria, essential for vocational and aptitude assessments. This document established guidelines for evaluating predictive evidence, requiring empirical demonstration of test-criterion relationships over time to ensure reliability in forecasting outcomes like job performance.[15] Building on these foundations in the 1960s and 1970s, Anne Anastasi's influential textbook Psychological Testing (3rd edition, 1968) underscored the role of predictive criteria in validating tests, advocating for longitudinal studies to assess how well measures anticipate real-world behaviors such as academic achievement or occupational success. Concurrently, Lee J. Cronbach and Paul E. Meehl's 1955 paper on construct validity indirectly shaped predictive subtypes by integrating them into broader validation frameworks, arguing that predictive evidence must align with theoretical constructs to avoid superficial correlations.[16] By the 1980s, predictive validity evolved toward a more unified concept, as reflected in the APA's 1985 Standards for Educational and Psychological Testing, which subsumed predictive evidence under a comprehensive validity umbrella, emphasizing its contribution to overall score interpretations rather than as an isolated category. This shift promoted multifaceted validation strategies, where predictive studies complemented other evidence types to support inferences in educational and psychological contexts.[17]Methods of Assessment
Criterion-Related Approaches
Criterion-related approaches to predictive validity involve empirical investigations that evaluate how well a test or measure forecasts future outcomes by correlating predictor scores with relevant criteria. These approaches emphasize the selection of appropriate criteria and the implementation of rigorous study designs to gather evidence of predictive power, as outlined in established psychometric standards.[1] Criterion selection is a foundational step, requiring the identification of future outcomes that are relevant, measurable, and aligned with the intended use of the predictor. Criteria must be reliable and representative of the target construct, such as job success metrics including productivity or task completion rates, to minimize extraneous variance and ensure meaningful predictions. Test developers and users bear responsibility for justifying criterion choices through job analysis or domain expertise, verifying that they capture the essence of the predicted behavior without contamination from irrelevant factors.[1][1] Study designs for predictive validity typically employ longitudinal cohorts, where the predictor is administered prior to observing the criterion, allowing time for outcomes to unfold naturally. This temporal separation distinguishes predictive from concurrent approaches and requires representative samples, detailed documentation of participant demographics, and controlled data collection to enhance generalizability and reduce biases like range restriction. Such designs facilitate the accumulation of validity evidence across diverse contexts, with cross-validation recommended to confirm stability over time.[1][1] Criteria in these approaches are categorized as hard or soft based on their objectivity. Hard criteria, also known as objective or nonjudgmental, derive from verifiable records with minimal subjective input, such as sales figures or production outputs, providing quantifiable and unbiased measures of performance. In contrast, soft criteria, or subjective and judgmental, depend on human evaluations like supervisor ratings or peer assessments, which may introduce rater bias but capture nuanced aspects of behavior. The choice between types depends on the construct's nature, with hard criteria preferred for precision and soft for comprehensive evaluation.[18][18] Validity coefficients, typically Pearson correlation coefficients between predictor and criterion scores, indicate the strength of predictive relationships, with interpretations guided by conventional benchmarks adjusted for context. A coefficient of r > 0.3 is generally considered moderate predictive power, signifying practically useful forecasting in fields like personnel selection, while values around 0.5 represent strong evidence, though observed coefficients are often attenuated by measurement error and require artifact corrections for accurate assessment. These interpretations must account for reliability estimates, sample size, and subgroup variations to avoid overgeneralization.[19][1]Statistical Techniques
The primary metric for evaluating predictive validity is the Pearson product-moment correlation coefficient (), which quantifies the linear relationship between scores on a predictor variable (e.g., a test) and a future criterion variable (e.g., job performance).[1] This coefficient ranges from -1 to +1, where values closer to 1 indicate stronger positive predictive relationships, 0 indicates no linear relationship, and negative values suggest inverse predictions; in psychometrics, values above 0.30 are often considered practically significant for predictive purposes.[1] The formula for is given by:
where are the predictor scores, are the criterion scores, and , are their respective means.[20]
Regression analysis extends this by deriving prediction equations to forecast criterion values from predictor scores, with simple linear regression serving as the foundational approach for single predictors.[1] The model takes the form , where is the predicted criterion score, is the predictor score, is the slope (equivalent to when both variables are standardized, indicating predictive strength), and is the intercept.[1] The slope reflects how much the criterion changes per unit increase in the predictor, allowing for practical applications like setting performance thresholds based on test results.[1]
For scenarios involving multiple predictors, multiple regression analysis is employed to model the combined predictive power, yielding an equation , where each represents the unique contribution of predictor while controlling for others.[1] This technique accounts for shared variance among predictors, providing a more comprehensive assessment of predictive validity than single-variable correlations.[1] To ensure generalizability beyond the sample, cross-validation is routinely applied, involving the estimation of model parameters on one dataset subset and testing predictions on an independent holdout subset, thereby mitigating overfitting and confirming the model's stability across new data.[1]
Significance testing accompanies these metrics to determine whether observed relationships are likely due to chance, typically using p-values from t-tests on or regression coefficients to test the null hypothesis that the population correlation .[20] Confidence intervals around or provide a range of plausible population values, offering insight into precision and reliability; for instance, narrow intervals indicate robust evidence for predictive validity.[1] Additionally, the coefficient of determination (or in multiple regression) measures the proportion of criterion variance explained by the predictor(s), serving as an effect size to contextualize practical importance—e.g., an (from ) implies 16% explained variance, a moderate effect in many psychometric applications.[20] These tests and effect sizes must be interpreted alongside sample size and potential artifacts like range restriction to avoid overestimating validity.[1]