CODESP - Cooperative Organization for the Development of Employee Selection Procedures Evaluating Selection Process Effectiveness Questions and Answers — Questions and Answers
Question 1: An HR analyst for a large state agency wants to demonstrate the financial return on investment of a new, validated selection system for IT professionals. The analyst plans to calculate the dollar value of the performance increase resulting from hiring higher-quality candidates, minus the cost of the new system. Which method is the analyst using to evaluate the selection process?
- Adverse impact analysis
- Content validation study
- Utility analysis (Correct answer)
- Inter-rater reliability assessment
Correct answer: Utility analysis
Utility analysis is a quantitative method used to estimate the financial and practical value of HR interventions, such as a new selection system. [5, 7] It specifically focuses on determining the return on investment by translating the benefits (e.g., increased job performance of new hires) into dollar terms and comparing them against the costs of the system. [7, 22]
Question 2: A personnel department has implemented a new selection test with a validity coefficient of .40. The current base rate of success for the job is 60% (i.e., 60% of current employees are successful). The selection ratio is low at .20 (20 hires for 100 applicants). What would Taylor-Russell tables be used to determine in this scenario?
- The likelihood that a single, specific candidate will be successful if hired.
- The proportion of new hires who are likely to be successful. (Correct answer)
- The test's reliability and internal consistency.
- Whether the test will cause adverse impact against a protected group.
Correct answer: The proportion of new hires who are likely to be successful.
Taylor-Russell tables are used to estimate the proportion of new hires who will be successful (the success ratio) given three key pieces of information: the test's criterion-related validity coefficient, the base rate of success in the current workforce, and the selection ratio. [1, 11] They demonstrate the incremental value a valid test provides over the existing success rate. [11, 24]
Question 3: A transit agency is evaluating its selection process for bus operators. Historically, 75% of operators hired prove to be successful performers after their probationary period. For the most recent hiring cycle, the agency received 500 applications for 50 open positions. In this context, what is the 'base rate'?
- 10%
- 50
- 500
- 75% (Correct answer)
Correct answer: 75%
The base rate is the percentage of current employees who are considered successful performers without the use of a new or particular selection system. [1, 12] In this scenario, the 75% success rate of previously hired operators represents the base rate. The selection ratio, by contrast, is the number of openings (50) divided by the number of applicants (500), which is 10%. [1, 12]
Question 4: A city's personnel department decides to raise the cut score on its written examination for librarians to be more selective. Which of the following is the MOST likely consequence of this action?
- The number of 'false positive' hiring errors will increase.
- The selection ratio will increase.
- The number of 'false negative' hiring errors will increase. (Correct answer)
- The base rate of success will decrease.
Correct answer: The number of 'false negative' hiring errors will increase.
Raising a cut score makes it harder to pass the test. This will likely screen out some candidates who would have been successful on the job, thereby increasing the number of 'false negatives'. [6, 23] Conversely, it tends to decrease the number of 'false positives' (unsuccessful performers who are hired) and decrease the selection ratio (fewer people are hired from the applicant pool). [4, 6, 23]
Question 5: A county implemented a multi-day assessment center for selecting department heads. A validity study shows it has excellent predictive validity. However, the cost is $10,000 per candidate, it requires six high-level staff members to administer, and candidates report high levels of stress and frustration with the lengthy process. When evaluating this selection system's overall effectiveness, what is the primary concern?
- Criterion-related validity
- Content validity
- Practicality and applicant reactions (Correct answer)
- Test-retest reliability
Correct answer: Practicality and applicant reactions
While the assessment center is valid, its high cost, resource-intensive administration, and negative impact on candidates make its overall effectiveness questionable. Evaluating a selection system must go beyond statistical validity to include practical considerations like cost, administrative feasibility, and applicant reactions, as these can determine if the system is sustainable and if it helps or harms the organization's reputation. [2, 9, 27]
Question 6: An agency's review of pass/fail data for a new physical ability test reveals that the pass rate for male candidates is 80%, while the pass rate for female candidates is 55%. According to the Uniform Guidelines, this disparity triggers a concern about adverse impact. What is the most critical next step in evaluating the legal defensibility of this test?
- Lowering the cut score for female candidates to equalize pass rates.
- Conducting a thorough validity study to demonstrate the test's job-relatedness. (Correct answer)
- Immediately discontinuing the use of the test for all future hiring.
- Surveying applicants about their perceptions of the test's fairness.
Correct answer: Conducting a thorough validity study to demonstrate the test's job-relatedness.
When a selection procedure results in adverse impact, the Uniform Guidelines on Employee Selection Procedures require the employer to demonstrate that the procedure is job-related and consistent with business necessity. [15] The most critical step to do this is to conduct a validity study (e.g., content, criterion, or construct validity) to gather evidence that the test measures essential job requirements. Simply adjusting scores or discontinuing the test without investigation are not the primary, legally-prescribed next steps.
CODESP - Cooperative Organization for the Development of Employee Selection Procedures Evaluating Selection Process Effectiveness Questions and Answers
An HR analyst for a large state agency wants to demonstrate the financial return on investment of a new, validated selection system for IT professionals.
The analyst plans to calculate the dollar value of the performance increase resulting from hiring higher-quality candidates, minus the cost of the new system.
Which method is the analyst using to evaluate the selection process?