CHSE Assessment and Evaluation Methods 2 — Questions and Answers
Question 1: Which assessment tool is MOST appropriate for evaluating a learner's real-time clinical decision-making during a high-fidelity simulation?
- Written multiple-choice examination
- Direct observation checklist with behavioral anchors (Correct answer)
- Post-simulation written reflective essay
- Peer-review portfolio assessment
Correct answer: Direct observation checklist with behavioral anchors
A behaviorally anchored observation checklist captures real-time clinical decisions and actions with standardized, objective criteria during simulation.
Question 2: In the context of simulation-based assessment, what does 'consequential validity' refer to?
- The accuracy of scoring rubrics
- The intended and unintended effects of assessment decisions on learners and programs (Correct answer)
- The degree to which an assessment predicts future performance
- The consistency of scores across multiple raters
Correct answer: The intended and unintended effects of assessment decisions on learners and programs
Consequential validity addresses both intended and unintended social and educational consequences arising from assessment use and score interpretation.
Question 3: A simulation educator wants to ensure that an OSCE station accurately measures IV insertion technique. Which type of validity is the PRIMARY concern?
- Predictive validity
- Construct validity
- Content validity (Correct answer)
- Concurrent validity
Correct answer: Content validity
Content validity ensures the assessment items adequately represent the full domain of the skill being measured, in this case IV insertion technique.
Question 4: Which of the following BEST describes formative assessment in simulation education?
- A high-stakes evaluation used to determine competency for promotion
- Ongoing feedback during learning designed to improve performance (Correct answer)
- A final summative judgment at the end of a training program
- A standardized test administered once per academic year
Correct answer: Ongoing feedback during learning designed to improve performance
Formative assessment provides ongoing, iterative feedback during the learning process to guide improvement rather than to make pass/fail decisions.
Question 5: When evaluating inter-rater reliability in simulation assessments, a kappa coefficient of 0.75 is obtained. How should this value be interpreted?
- Poor agreement, requiring immediate rubric revision
- Slight agreement, suggesting rater training is needed
- Substantial agreement, generally acceptable for most assessments (Correct answer)
- Perfect agreement, no further calibration needed
Correct answer: Substantial agreement, generally acceptable for most assessments
A kappa of 0.61–0.80 is classified as substantial agreement and is generally considered acceptable for clinical skills assessments.
Question 6: A simulation program uses the same test both before and after a training intervention to measure learning gains. This approach is BEST described as:
- Concurrent validity testing
- Pre-test/post-test design (Correct answer)
- Criterion-referenced evaluation
- Norm-referenced assessment
Correct answer: Pre-test/post-test design
A pre-test/post-test design measures the same construct before and after instruction to quantify learning gains attributable to the intervention.
Question 7: Which evaluation model is MOST commonly referenced when assessing the overall effectiveness of a simulation training program across multiple levels, including reaction, learning, behavior, and results?
- Kirkpatrick's Four-Level Model (Correct answer)
- Miller's Pyramid
- Bloom's Taxonomy
- Kolb's Experiential Learning Cycle
Correct answer: Kirkpatrick's Four-Level Model
Kirkpatrick's Four-Level Model evaluates training programs at the levels of reaction, learning, behavior change, and organizational results.
CHSE Assessment and Evaluation Methods 2
Which assessment tool is MOST appropriate for evaluating a learner's real-time clinical decision-making during a high-fidelity simulation?