CODESP Test Validation and Reliability 3 — Questions and Answers
Question 1: A job analysis identifies five critical task clusters for a position. A test that covers only two of those clusters likely lacks:
- Predictive validity
- Content validity (Correct answer)
- Construct validity
- Criterion-related validity
Correct answer: Content validity
Content validity requires adequate coverage of all critical KSAs and tasks identified in the job analysis.
Question 2: The attenuation of a validity coefficient due to restriction of range refers to:
- A coefficient that is artificially inflated because of a broad range of scores
- A coefficient that is artificially deflated because the sample lacks score variability (Correct answer)
- An error in the criterion measure only
- Measurement error in the predictor test only
Correct answer: A coefficient that is artificially deflated because the sample lacks score variability
Range restriction occurs when hired employees represent a narrow band of scores, which deflates observed validity coefficients.
Question 3: Which statistic is most commonly used to assess internal consistency reliability for tests with polytomous (multi-point) items?
- Kuder-Richardson 20 (KR-20)
- Cronbach's alpha (Correct answer)
- Spearman-Brown formula
- Cohen's kappa
Correct answer: Cronbach's alpha
Cronbach's alpha is appropriate for items with multiple response categories, while KR-20 is for dichotomous items.
Question 4: A validity generalization (VG) study concludes that a test has consistent validity across multiple settings. This finding supports:
- The need for local validation in every organization
- The transportability of test validity to new settings without local studies (Correct answer)
- That criterion-related validity is unnecessary
- That content validity supersedes criterion validity
Correct answer: The transportability of test validity to new settings without local studies
Validity generalization supports using existing validity evidence to infer a test's validity in a new setting without conducting a full local study.
Question 5: When evaluating construct validity, convergent evidence is demonstrated when:
- The test correlates highly with theoretically unrelated measures
- The test does NOT correlate with measures of similar constructs
- The test correlates highly with measures of theoretically similar constructs (Correct answer)
- The test predicts job performance six months later
Correct answer: The test correlates highly with measures of theoretically similar constructs
Convergent evidence shows that measures of the same or related constructs correlate substantially with each other.
Question 6: The Uniform Guidelines on Employee Selection Procedures recommend that a content validity strategy is most defensible when the selection procedure measures:
- General cognitive ability
- Observable work behaviors or knowledge used on the job (Correct answer)
- Personality traits predictive of job success
- Motivational factors assessed by interviews
Correct answer: Observable work behaviors or knowledge used on the job
The Uniform Guidelines indicate content validity is most appropriate for tests measuring observable work behaviors or directly job-related knowledge.
Question 7: Inter-rater reliability is MOST critical to establish when:
- A multiple-choice test is machine-scored
- Structured interviews or work samples involve human judgment in scoring (Correct answer)
- A standardized test is administered online
- Scores are derived from a biodata inventory
Correct answer: Structured interviews or work samples involve human judgment in scoring
Inter-rater reliability ensures that different evaluators assign consistent scores when human judgment is involved in the scoring process.
CODESP Test Validation and Reliability 3
A job analysis identifies five critical task clusters for a position.
A test that covers only two of those clusters likely lacks: