NCMHCE Research and Program Evaluation 4 — Questions and Answers
Question 1: A counselor wants to evaluate whether a new group therapy program reduced clients' depression scores over 12 weeks. Which design best controls for threats to internal validity?
- Single-group pretest-posttest design
- Randomized controlled trial with a waitlist control group (Correct answer)
- Case study analysis of three participants
- Retrospective chart review without a comparison group
Correct answer: Randomized controlled trial with a waitlist control group
A randomized controlled trial with a control group controls for history, maturation, and regression threats that plague single-group designs.
Question 2: In a program evaluation, the evaluator finds a statistically significant result (p = .03) but a Cohen's d of 0.10. What is the most accurate interpretation?
- The intervention is both statistically and practically significant
- The result is statistically significant but the effect size is negligible (Correct answer)
- The p-value is too high to trust the finding
- A d of 0.10 always indicates a medium effect
Correct answer: The result is statistically significant but the effect size is negligible
Statistical significance reflects sample size, while Cohen's d < 0.20 indicates a negligible practical effect regardless of the p-value.
Question 3: Which sampling method would be most appropriate when a researcher needs to ensure representation from each of five distinct cultural communities in a city?
- Simple random sampling
- Convenience sampling
- Stratified random sampling (Correct answer)
- Snowball sampling
Correct answer: Stratified random sampling
Stratified random sampling divides the population into subgroups and randomly samples from each, ensuring proportional or equal representation.
Question 4: A counselor reviewing a needs assessment report notices that data were collected only from clients who completed treatment. This is an example of which bias?
- Response bias
- Attrition bias (survivor bias) (Correct answer)
- Social desirability bias
- Instrumentation bias
Correct answer: Attrition bias (survivor bias)
Attrition (survivor) bias occurs when only treatment completers are assessed, systematically excluding dropouts who may have worse outcomes.
Question 5: Which statistical test is most appropriate to compare pre-treatment and post-treatment anxiety scores from the same group of 30 clients?
- Independent samples t-test
- Chi-square test
- Paired samples t-test (Correct answer)
- One-way ANOVA
Correct answer: Paired samples t-test
The paired samples t-test is designed for comparing two measurements from the same participants, controlling for individual differences.
Question 6: A program evaluator is asked to determine whether a school counseling intervention should be continued, modified, or discontinued. This is best described as what type of evaluation?
- Formative evaluation
- Needs assessment
- Summative evaluation (Correct answer)
- Process evaluation
Correct answer: Summative evaluation
Summative evaluation judges overall program worth or merit to inform continuation, modification, or termination decisions at program's end.
Question 7: Which of the following best illustrates a Type II error in mental health research?
- Concluding a treatment works when it actually does not
- Failing to detect a true treatment effect because the study was underpowered (Correct answer)
- Using an invalid measure that inflates scores
- Publishing results with a p-value of .001
Correct answer: Failing to detect a true treatment effect because the study was underpowered
A Type II error (false negative) occurs when a researcher fails to reject a null hypothesis that is actually false, often due to insufficient statistical power.
NCMHCE Research and Program Evaluation 4
A counselor wants to evaluate whether a new group therapy program reduced clients' depression scores over 12 weeks.
Which design best controls for threats to internal validity?