CODESP Test Validation and Reliability 4 — Questions and Answers
Question 1: A test developer applies the Spearman-Brown prophecy formula. What is the primary purpose of this formula?
- To correct for criterion unreliability
- To estimate the reliability of a lengthened or shortened test (Correct answer)
- To assess differential item functioning
- To calculate the standard error of estimate
Correct answer: To estimate the reliability of a lengthened or shortened test
The Spearman-Brown formula predicts how reliability will change as a function of increasing or decreasing test length.
Question 2: Which scenario best illustrates the problem of criterion contamination?
- Supervisors rate employee performance without knowing the employee's test score
- Supervisors know an employee's test score before providing performance ratings (Correct answer)
- The criterion measure is collected six months after hiring
- Two raters independently score the same work sample
Correct answer: Supervisors know an employee's test score before providing performance ratings
Criterion contamination occurs when supervisors' knowledge of test scores influences their performance ratings, biasing the criterion.
Question 3: In item response theory (IRT), which parameter describes how steeply an item distinguishes between higher and lower ability test takers?
- The b-parameter (difficulty)
- The c-parameter (pseudo-guessing)
- The a-parameter (discrimination) (Correct answer)
- The d-parameter (differential functioning)
Correct answer: The a-parameter (discrimination)
The a-parameter in IRT reflects item discrimination — how sharply the item separates individuals with different ability levels.
Question 4: A selection test shows high validity for a sample of incumbents but poor validity for a sample of applicants. The MOST likely explanation is:
- The test lacks content validity
- Range restriction in the incumbent sample inflated the coefficient (Correct answer)
- The criterion measure is unreliable
- Construct irrelevant variance affected the applicant sample
Correct answer: Range restriction in the incumbent sample inflated the coefficient
Incumbent samples are often range-restricted because low performers have already left, which can inflate validity coefficients relative to applicant samples.
Question 5: Consequential validity refers to the idea that validation should include evidence about:
- Correlation with job performance criteria only
- The intended and unintended social consequences of test use (Correct answer)
- Whether the test was developed by an internal team or consultants
- The test's face validity as perceived by applicants
Correct answer: The intended and unintended social consequences of test use
Consequential validity (Messick's framework) incorporates evidence about the ethical and social impact of testing decisions.
Question 6: When a test battery includes both a cognitive ability test and a structured interview, and the two are combined into a total score, this practice is known as:
- Multiple hurdle selection
- Compensatory combination (Correct answer)
- Differential weighting
- Banding
Correct answer: Compensatory combination
Compensatory combination allows high scores on one predictor to offset low scores on another when computing a composite score.
Question 7: A test that consistently measures a construct that is NOT the intended construct is said to have high:
- Construct-relevant variance
- Construct-irrelevant variance (Correct answer)
- Criterion deficiency
- Criterion contamination
Correct answer: Construct-irrelevant variance
Construct-irrelevant variance occurs when test scores reflect factors unrelated to the target construct, threatening construct validity.
CODESP Test Validation and Reliability 4
A test developer applies the Spearman-Brown prophecy formula.
What is the primary purpose of this formula?