Evaluating Statistical Claims: Observational Studies and Experiments Flashcards

A researcher finds that cities with more hospitals per capita have higher death rates than cities with fewer hospitals per capita. A news headline reads: 'Hospitals Increase Risk of Death.' Which of the following best explains why this conclusion is flawed?

Answer: Confounding variables, such as population age and pre-existing illness rates, likely explain the association.

This is a classic confounding variable problem. Sicker and older populations are concentrated in areas with more hospitals — the hospitals are responding to need, not causing death. The association is driven by a lurking variable (illness severity), not a causal relationship between hospital presence and mortality.

In a randomized controlled experiment, participants are randomly assigned to take either a new supplement or a placebo. At the end of the study, those who took the supplement reported significantly better sleep. However, 23% of participants in the supplement group dropped out before the study ended, compared to 4% in the placebo group. What is the most serious threat to the validity of this study's conclusion?

Answer: Differential attrition may have biased the results by removing participants who responded poorly.

Differential attrition — when dropout rates differ significantly between groups — is a major threat to internal validity. If participants who dropped out of the supplement group did so because the supplement worsened their sleep, the remaining participants are a self-selected group of those who tolerated or benefited from it, biasing the results in favor of the supplement.

A study reports that students who eat breakfast score an average of 12 points higher on standardized tests than students who skip breakfast. The study was observational, with a sample of 2,400 students drawn from 15 schools. Which of the following conclusions is best supported by these data?

Answer: There is an association between eating breakfast and higher test scores, but causation cannot be established.

Observational studies can identify associations but cannot establish causation because participants are not randomly assigned to conditions. Socioeconomic status, for example, may influence both whether a student eats breakfast and their academic performance. A 2,400-student sample across 15 schools is reasonably large; the issue is the study design, not the sample size.

Researchers want to test whether a new tutoring program improves SAT math scores. They recruit 200 student volunteers and allow students to choose whether to enroll in the tutoring program or not. After 8 weeks, the tutoring group scores 40 points higher on average. Which of the following design flaws most directly undermines a causal interpretation?

Answer: Self-selection bias means students who chose tutoring may differ systematically from those who did not.

Because students chose their own group, the two groups may differ in motivation, prior preparation, or access to other resources — all of which could independently raise scores. This self-selection bias confounds the treatment effect and prevents causal claims. Random assignment is the mechanism that controls for these pre-existing differences.

A well-designed, double-blind randomized experiment with 500 participants finds that a new medication reduces headache frequency by a statistically significant margin (p = 0.03). A critic argues the result is not practically meaningful. Which of the following, if true, would most strongly support the critic's position?

Answer: The medication reduced headaches by an average of 0.2 episodes per month compared to placebo.

Statistical significance (p < 0.05) indicates that a result is unlikely to be due to chance alone, but it does not measure practical or clinical significance. A reduction of only 0.2 headaches per month — even if real — may be too small to meaningfully improve patients' lives. Effect size, not just p-value, determines practical relevance.

In a study of 1,000 adults, researchers find that people who drink at least two cups of coffee per day are 30% less likely to develop a certain liver condition. The researchers controlled for age, BMI, and alcohol consumption in their analysis. A follow-up randomized experiment assigns participants to drink coffee or a coffee-flavored decaf beverage for two years and finds no significant difference in liver condition rates. What is the most reasonable interpretation of these two studies taken together?

Answer: The apparent protective effect of coffee in the observational study was likely explained by a confounding variable not fully captured by the controlled variables.

When a controlled experiment fails to replicate an association found in an observational study, the most likely explanation is that the original association was driven by a confounding variable the researchers did not fully control for. Even controlling for age, BMI, and alcohol, other unmeasured variables (diet quality, exercise habits, genetic factors) may have explained the link. The experiment's null result suggests the coffee itself was not the active cause.