Evaluating Statistical Claims: Observational Studies and Experiments Flashcards

A university randomly assigns 60 first-year students to peer mentoring and 60 to no mentoring. Those with mentors have higher GPA after one year. A dean applies this to all students at the university. What is the main limitation of this generalization?

Answer: The 120 students may not be representative of all first-year students at the university.

The causal conclusion is valid within the study, but with only 120 students from potentially non-representative groups, generalizing to all students at the university requires caution.

Researchers study whether a new teaching strategy improves reading comprehension by observing classrooms that voluntarily adopt the strategy versus those that do not. What is the primary limitation of this study?

Answer: Because teachers self-selected the strategy, more motivated or skilled teachers may have adopted it, confounding results.

Self-selection bias means that teachers who volunteer may be systematically different from those who don't, making it unclear whether the strategy or the teacher quality is responsible for better outcomes.

A well-designed randomized experiment conducted on college athletes in one state shows that a protein supplement improves sprint times. A coach in a different state wants to apply this finding to high school students. What concern is most valid?

Answer: High school students may differ from college athletes in age, physiology, and training level, limiting generalizability.

Even though the experiment established causation within the study group, the finding may not generalize to high school students who differ physiologically and in training status from college athletes.

An observational study shows that people who drink red wine moderately have fewer cardiovascular events than abstainers. A doctor recommends wine to all patients to protect their heart. Which of the following best explains why this recommendation is premature?

Answer: The association may be explained by confounders; moderate drinkers may also follow healthier overall lifestyles.

Moderate drinkers may exercise more, eat better, or have lower baseline stress — confounders that could explain the lower cardiovascular event rate rather than the wine itself.

A study randomly assigns 50 households to receive a water filter and 50 households to not receive one. After 6 months, the filter group reports fewer gastrointestinal illnesses. Which of the following is a valid conclusion?

Answer: In households similar to those in the study, water filters likely cause fewer gastrointestinal illnesses.

Random assignment supports a causal conclusion within the study; however, the result applies to households similar to those studied, not universally.

A survey finds that people who own dogs walk more steps per day than non-dog-owners. The researcher concludes: 'Dog ownership increases daily step count.' What type of evidence would strengthen this causal claim?

Answer: A randomized experiment in which some participants are randomly assigned a dog

Only random assignment to owning or not owning a dog would eliminate confounding (e.g., active people choosing dogs) and allow a causal conclusion.

A large randomized experiment with diverse participants from 20 states finds that a new reading intervention improves comprehension scores. A state education board generalizes this to all students in the country. Which concern about generalizability is least valid?

Answer: The study used random assignment, which undermines generalizability.

Random assignment has nothing to do with generalizability; it supports causal inference. The legitimate concerns involve sample representativeness and contextual differences.