P-Hacking (Data Dredging) — When Logic Wears a Disguise

P-hacking occurs when researchers repeatedly analyze data using different methods, variable selections, or subgroup divisions until a statistically significant p-value (typically below 0.05) is found. This exploitation of researcher degrees of freedom inflates false-positive rates far beyond the nominal significance level. The practice can be intentional or unconscious, driven by publication incentives that reward significant findings.

Also known as: data dredging, significance chasing, researcher degrees of freedom exploitation

How It Works

With a 5% significance threshold, testing 20 independent hypotheses gives roughly a 64% chance of at least one false positive. Audiences typically see only the reported result, not the full search process.

A Classic Example

A pharmaceutical researcher tests a new supplement against 20 different health outcomes. One outcome (toenail growth rate) yields p < 0.05. The paper is published with the title 'Supplement X significantly improves toenail growth' without mentioning the 19 non-significant tests.

More Examples

A marketing researcher collects data on a new ad campaign and finds no significant effect on overall sales. They then slice the data by age group, region, time of day, device type, and day of week until they find that women aged 35–44 in the Midwest who saw the ad on a Tuesday showed p = 0.04 — and report this as a key finding.

A nutrition scientist studies whether a diet intervention reduces body weight, cholesterol, and blood pressure. None reach significance. They then test 15 additional biomarkers and find one — fasting insulin — with p = 0.049, which becomes the headline result of the published paper.

Where You See This in the Wild

P-hacking is rampant in social psychology and biomedical research, contributing to the replication crisis. Journals like PLOS ONE now require pre-registration to combat it.

How to Spot and Counter It

Demand pre-registration of hypotheses and analysis plans. Apply corrections for multiple comparisons such as Bonferroni or Benjamini-Hochberg, and ask how many tests were conducted in total.

The Takeaway

The P-Hacking (Data Dredging) is one of those reasoning errors that sounds perfectly logical at first glance. That's what makes it dangerous — it wears the costume of valid reasoning while smuggling in a broken conclusion. The best defense? Slow down and ask: does this conclusion actually follow from these premises, or am I just connecting dots that happen to be near each other?

Next time someone presents you with an argument that "just makes sense," check the structure. The feeling of logic is not the same as logic itself.