Garden of Forking Paths — When Logic Wears a Disguise

The Garden of Forking Paths describes how researchers, even without malicious intent, make numerous small analytical decisions (how to define variables, which outliers to exclude, which covariates to include, when to stop collecting data) that collectively inflate the false positive rate. Unlike p-hacking (deliberate fishing for significance), this can happen unconsciously: each individual decision seems reasonable, but the cumulative effect of many 'reasonable' choices made while looking at the data dramatically increases the chance of finding a spurious result. Named by Andrew Gelman and Eric Loken (2013) after a Borges short story.

Also known as: Researcher Degrees of Freedom, Analytical Flexibility

How It Works

Each decision point is individually defensible, making it impossible to point to a single 'error.' The researcher isn't lying or cheating — they're making judgment calls that happen to converge on a publishable result. The problem is invisible because only the final analysis is reported, not the dozens of alternative analyses that were implicitly considered and rejected.

A Classic Example

A psychology study finds that people who eat chocolate score higher on a creativity test. But the researchers made dozens of decisions: which creativity test to use, whether to control for age, how to define 'regular chocolate consumption,' whether to include participants who didn't finish, which statistical test to apply. With different but equally defensible choices at each fork, the result might vanish.

More Examples

A replication team follows the original paper's methods exactly but still finds no effect. The original researchers made dozens of small decisions — data collection timing, exact wording of instructions, analysis software — that shaped the result but weren't reported. The forking paths are invisible in the published article.

An economics paper finds that a minimum wage increase had no effect on employment in treated counties. The result holds with their chosen control counties, their chosen time window, and their model specification. Three independent teams reanalysing the same public data with different (equally valid) analytical choices find effects ranging from −8% to +4%.

Where You See This in the Wild

Gelman and Loken showed that many famous psychology findings (power poses, ego depletion, social priming) likely resulted from this process. The replication crisis in social science is partly attributable to forking paths: original studies found effects through one specific analytical path that failed to replicate when others tried different (but equally valid) paths.

How to Spot and Counter It

Pre-register your analysis plan before collecting data. Conduct sensitivity analyses showing how results change with different analytical choices. Report all decisions made during analysis, not just the final pipeline. Use multiverse analysis to map results across all reasonable analytical paths.

The Takeaway

The Garden of Forking Paths is one of those reasoning errors that sounds perfectly logical at first glance. That's what makes it dangerous — it wears the costume of valid reasoning while smuggling in a broken conclusion. The best defense? Slow down and ask: does this conclusion actually follow from these premises, or am I just connecting dots that happen to be near each other?

Next time someone presents you with an argument that "just makes sense," check the structure. The feeling of logic is not the same as logic itself.