TAPAS Test Validity and Reliability 5 — Questions and Answers
Question 1: The concept of 'validity generalization' applied to TAPAS suggests that:
- A test validated in one military branch cannot be used in another
- Validity evidence accumulated across settings supports use in new, similar contexts (Correct answer)
- Each new administration site must conduct an independent criterion study
- Validity coefficients remain constant regardless of range restriction
Correct answer: Validity evidence accumulated across settings supports use in new, similar contexts
Validity generalization (Schmidt & Hunter) shows that observed variation in validity coefficients across studies is largely due to statistical artifacts, allowing evidence from prior studies to support new applications.
Question 2: Range restriction in TAPAS validity studies most commonly occurs because:
- The test contains too few items to capture score variability
- Only individuals who scored high enough to be selected are included in the criterion sample (Correct answer)
- Adaptive item selection narrows the range of personality scores obtained
- Military performance ratings are measured on a limited ordinal scale
Correct answer: Only individuals who scored high enough to be selected are included in the criterion sample
When only selected (high-scoring) applicants enter the criterion sample, the score distribution is truncated, which attenuates observed validity coefficients relative to the true population value.
Question 3: A TAPAS score that is highly reliable but measures the wrong construct exemplifies:
- High reliability guaranteeing high validity
- The distinction that reliability is necessary but not sufficient for validity (Correct answer)
- Concurrent validity with a misspecified criterion
- Adequate internal consistency overcoming construct bias
Correct answer: The distinction that reliability is necessary but not sufficient for validity
Reliability is a prerequisite for validity—a score must be consistent before it can validly measure anything—but consistent measurement of the wrong thing is reliable yet invalid.
Question 4: When conducting a TAPAS item review panel to establish content validity, the panel should ideally consist of:
- Only psychometricians with IRT expertise
- Subject-matter experts familiar with the construct and the target population (Correct answer)
- Military applicants who have not yet taken the test
- Statisticians who can compute inter-rater agreement coefficients
Correct answer: Subject-matter experts familiar with the construct and the target population
Content validity panels require subject-matter experts who understand both the psychological construct (e.g., conscientiousness) and the job context (military service) to judge item representativeness and relevance.
Question 5: In a TAPAS validity study, the disattenuation formula is applied to correct an observed correlation for:
- Range restriction in the applicant sample
- Measurement error in both the predictor and criterion (Correct answer)
- Differential item functioning across demographic groups
- Non-normality of the score distribution
Correct answer: Measurement error in both the predictor and criterion
The disattenuation (correction for attenuation) formula adjusts an observed correlation upward to estimate the true-score correlation by removing the dampening effect of measurement error in both variables.
Question 6: Which scenario would provide the strongest evidence that TAPAS scores have utility beyond selection and should inform personnel classification decisions?
- TAPAS scores correlate equally with performance in all military occupational specialties
- Different TAPAS profiles predict success differentially across distinct military job families (Correct answer)
- The test shows the same mean scores regardless of which MOS applicants are pursuing
- TAPAS reliability is above 0.90 for every personality dimension measured
Correct answer: Different TAPAS profiles predict success differentially across distinct military job families
If specific personality profiles predict success in some jobs better than others (differential prediction), TAPAS can improve classification by matching individuals to the roles where they are most likely to thrive.
Question 7: A cross-validation study of TAPAS shrinks the initial validity coefficient considerably when applied to a new sample. This shrinkage is best explained by:
- Construct underrepresentation in the original item pool
- Capitalization on chance in the original model, which does not replicate (Correct answer)
- Insufficient test-retest reliability across the two samples
- Differential item functioning between the derivation and validation samples
Correct answer: Capitalization on chance in the original model, which does not replicate
Shrinkage occurs because a model optimized on one sample fits sample-specific noise; cross-validation reveals the true, lower validity by applying the model to new data where that noise is absent.
TAPAS Test Validity and Reliability 5
The concept of 'validity generalization' applied to TAPAS suggests that: