Publication Bias (File Drawer Problem) — When Numbers Lie

Has this ever happened to you? Ten labs independently test whether listening to Mozart improves spatial reasoning.

Also known as: file drawer problem, positive results bias, reporting bias

What's Actually Happening

Publication bias is the systematic tendency for journals and researchers to preferentially publish studies with positive or statistically significant results, while studies with null or negative findings remain unpublished in the 'file drawer.' This distorts the available body of evidence, making effects appear larger and more consistent than they truly are. Meta-analyses based on published literature inherit this bias, potentially validating interventions that are ineffective.

Incentive structures in academia reward novel positive findings. Null results are seen as uninteresting and are harder to publish, creating a systematic filter that favors one type of outcome.

Real Talk: You See This Every Day

Ten labs independently test whether listening to Mozart improves spatial reasoning. Three labs find a significant positive effect and publish. Seven labs find no effect and do not publish. A meta-analysis of the published studies concludes that the 'Mozart effect' is robust and significant.

Publication bias has been extensively documented in pharmaceutical research (negative drug trials hidden), psychology (inflated effect sizes), and educational interventions.

Your BS Detector

Consult pre-registration databases (ClinicalTrials.gov, OSF). Use funnel plots and statistical tests for publication bias (Egger's test). Support journals that publish null results (e.g., Journal of Articles in Support of the Null Hypothesis).

✓ Who collected this data, and why?
✓ Is the sample big enough and fair?
✓ Could there be another explanation?

The Challenge

Next time someone throws a statistic at you — in class, online, in the news — don't just accept it. Ask: what's missing from this picture?

Part of the TellDear Teen Book — criticalthinking.guide