Evaluating Statistical Claims: Observational Studies and Experiments Flashcards

A researcher studying the effect of meditation on stress conducts a randomized experiment. Participants are randomly assigned to either a meditation group or a control group. After 8 weeks, the meditation group shows significantly lower cortisol levels. A critic argues that the results are invalid because participants knew which group they were in. Which statistical concept does this criticism most directly address?

Answer: Lack of blinding, which may introduce response bias or placebo effects that confound the treatment effect

The critic's concern is specifically about participants knowing their group assignment — a blinding issue. When participants know they're receiving a treatment, they may behave differently or report feeling better due to the placebo effect, not the treatment itself. This is distinct from confounding (which random assignment controls for), sample size, or external validity. The study already used randomization, so confounding is largely addressed.

Two studies examine whether drinking coffee reduces the risk of Type 2 diabetes. Study A is a 10-year prospective cohort study following 50,000 adults who self-report coffee consumption. Study B is a randomized controlled trial (RCT) with 200 participants assigned to drink coffee or not for 6 months. Study A finds a strong association (RR = 0.72, p < 0.001); Study B finds no significant effect (p = 0.43). Which conclusion is most statistically justified?

Answer: Study A's association may reflect confounding — coffee drinkers may share other protective lifestyle habits — while Study B's causal inference is stronger despite lower power

The key tension here is internal validity vs. statistical power. Study A has high statistical power due to its large sample but is vulnerable to confounding — coffee drinkers may also exercise more, eat better, etc. Study B eliminates confounding through randomization (stronger causal inference) but has low power (only 200 participants for a 6-month period may be insufficient to detect a modest long-term effect). The most justified conclusion acknowledges that the association in Study A may not be causal, and that Study B's design is superior even if underpowered.

In a study on whether a new teaching method improves math scores, students at one school use the new method while students at another school use the traditional method. At year's end, the new-method school scores 12 points higher on average. A statistician warns this design is fundamentally flawed. What is the PRIMARY flaw?

Answer: Students were not randomly assigned to schools, so school-level differences (resources, demographics, teacher quality) confound the treatment effect

The fundamental flaw is the absence of random assignment — this is a quasi-experiment, not a true experiment. Students self-selected (or were assigned) to schools, meaning the schools likely differ in ways that also affect math scores (socioeconomic status, school funding, prior achievement levels). No statistical test can fully untangle teaching method effects from these pre-existing school differences. While the other options identify real issues, the primary structural flaw is confounding due to non-random assignment.

A survey finds that among adults who eat breakfast daily, 23% report high life satisfaction, compared to 41% among those who skip breakfast. The researchers conclude that skipping breakfast causes higher life satisfaction. Which of the following identifies the most critical error in this reasoning?

Answer: The direction of causality is reversed — it is equally plausible that people with higher life satisfaction have lifestyles that lead them to skip breakfast, not the other way around

This is a classic reverse causality problem. The researchers assumed that breakfast habits influence life satisfaction, but the data are consistent with the opposite: people who are already happy may have more energetic, social lifestyles that involve brunches or intermittent fasting. Observational data alone cannot establish the direction of causation. The other choices raise minor methodological points but do not address the fundamental logical flaw in the causal claim.

Researchers want to test whether a mindfulness app reduces anxiety in college students. They recruit 300 volunteers and randomly assign 150 to use the app for 4 weeks and 150 to a waitlist control. At the end, 30% of the app group dropped out versus 5% in the control group. The remaining participants in the app group show significantly lower anxiety. What is the most serious threat to the validity of this conclusion?

Answer: Differential attrition: those who dropped out of the app group likely had worse anxiety outcomes, meaning the remaining sample is systematically biased toward those who benefited

The 30% dropout rate in the app group versus 5% in the control group creates a critical differential attrition problem. Those who dropped out likely found the app unhelpful or experienced worsening anxiety — by excluding them, the final analysis only includes the 'survivors' who benefited. This systematically inflates the apparent effectiveness of the app. Intent-to-treat analysis (keeping all randomized participants in the analysis) is the standard way to address this. The other issues are real but secondary to this bias.

A county health department notices that neighborhoods with more fast-food restaurants have higher rates of childhood obesity. To test causality, they propose placing fast-food restaurants in low-obesity neighborhoods and monitoring obesity rates over 5 years. A biostatistician objects, calling this plan both unethical and statistically insufficient for a different reason. What statistical limitation would remain even if the ethical issue were ignored?

Answer: Without random assignment of which specific neighborhoods receive restaurants, pre-existing neighborhood differences (income, park access, food culture) remain as confounders

Even if researchers added fast-food restaurants to some neighborhoods, without random selection of which neighborhoods receive them, the comparison is still confounded. Neighborhoods where health officials choose to place restaurants likely differ from control neighborhoods in income, existing food access, population density, and other factors. True experimental inference requires that the treatment (restaurant placement) is assigned randomly, not chosen based on existing characteristics. The ecological fallacy (answer C) is a related concept but refers to incorrectly attributing group-level patterns to individuals — a different issue from this confounding problem.