Structured Oral Examination Design Flashcards

In a structured oral examination, what is the primary purpose of using an 'anchored rating scale'?

Answer: To provide behaviorally defined benchmark responses at each score level for consistent evaluation

Anchored rating scales define specific behavioral indicators at each score point, reducing subjectivity and improving inter-rater reliability.

Which approach best reduces the 'similar-to-me' bias in structured oral examination panels?

Answer: Training raters on common biases and requiring independent scoring

Rater training on cognitive biases combined with independent scoring before any discussion is the most effective mitigation strategy.

A CODESP-developed structured oral examination uses 'probing questions.' What is their appropriate use?

Answer: To clarify ambiguous responses or elicit additional relevant information within the same competency

Probing questions are pre-approved follow-ups used to draw out more complete information within the same competency area, not to introduce new topics.

When establishing content validity for a structured oral examination, which step is most critical?

Answer: Linking each examination question directly to KSAOs identified in a job analysis

Content validity depends on a demonstrable linkage between exam questions and job-relevant KSAOs identified through systematic job analysis.

In CODESP structured oral exam administration, why are panelists typically prohibited from sharing scores until all panelists have scored independently?

Answer: To prevent anchoring bias where early scores influence subsequent raters

Sequential scoring discussions create anchoring effects; independent scoring preserves the statistical independence needed for reliable inter-rater agreement.

Which factor most directly threatens the reliability of a structured oral examination administered across multiple sites on different days?

Answer: Inconsistent question reading, timing, or probing practices across different panels

Procedural inconsistencies in how questions are presented and probed across panels directly undermine standardization and score comparability.

Under EEO guidelines relevant to CODESP exams, adverse impact in a structured oral examination is most appropriately evaluated using:

Answer: The 80% (four-fifths) rule applied to pass rates of protected groups versus the highest-scoring group

The UGESP four-fifths rule compares selection rates: if a protected group's rate is less than 80% of the top group's rate, adverse impact is indicated.