Type I Error / False Positive: Seeing What Isn't There

In 1999, Sally Clark was convicted of murdering her two infant sons. The prosecution's centrepiece was statistical testimony from paediatrician Sir Roy Meadow, who told the jury that the probability of two children in the same family dying of sudden infant death syndrome (SIDS) was 1 in 73 million. The jury convicted. The statistic was wrong — it ignored that SIDS has familial clustering and violated a basic rule of probability — and Clark spent three years in prison before her conviction was overturned. A catastrophic false positive, built on a false positive in the statistics.

What Is a Type I Error?

In the framework of hypothesis testing, a Type I error occurs when you reject the null hypothesis even though it is true. The null hypothesis is typically the default position — "this drug has no effect," "these two groups don't differ," "the defendant is innocent." Rejecting it when it's actually correct means concluding that something is real when it isn't.

In everyday language, a Type I error is a false positive: you detect a signal that isn't there. The alarm sounds when there's no fire. The test says positive when the patient is healthy. The model flags a transaction as fraudulent when it's legitimate. The jury convicts an innocent person.

The probability of a Type I error is called alpha (α) — the significance level. In most scientific research, α is set at 0.05, meaning researchers accept a 5% chance of falsely rejecting the null hypothesis in any given test. This threshold is conventional, not sacred, and its implications are frequently misunderstood. A p-value below 0.05 does not mean the finding is true; it means that if the null hypothesis were true, you'd expect to see results at least this extreme only 5% of the time by chance.

The Justice System: Convicting the Innocent

The criminal law embodies a deliberate stance on Type I and Type II errors. The standard "beyond reasonable doubt" is designed to minimise false positives — wrongful convictions of innocent people — even at the cost of more false negatives (guilty people acquitted). The asymmetry reflects a moral judgment: convicting an innocent person is a worse error than letting a guilty one go free.

The Sally Clark case illustrates how the justice system can fail when statistical reasoning goes wrong. Sir Roy Meadow's "1 in 73 million" figure multiplied the probability of one SIDS death by itself — assuming the two events were independent, which they weren't. The Royal Statistical Society issued a public statement criticising the evidence; the Court of Appeal eventually quashed the conviction. But Clark spent years imprisoned and died in 2007, reportedly never recovering from the ordeal.

DNA exoneration data reveals the systematic presence of false positives in the justice system. The Innocence Project in the United States has secured the exoneration of more than 375 wrongfully convicted people since 1992. The most common contributing factors: eyewitness misidentification (present in over 69% of cases), unvalidated forensic science, false confessions, and informant testimony. Each represents a mechanism for generating Type I errors — detecting guilt where none exists.

Eyewitness testimony is particularly striking. Research by Elizabeth Loftus and others has demonstrated that human memory is reconstructive, not reproductive — it assembles a plausible story from fragments rather than playing back a recording. Memory is systematically distorted by post-event information, suggestion, leading questions, and the passage of time. In police lineups, witnesses often identify the person who looks most like the perpetrator among those present rather than the actual perpetrator. The confident identification — "that's him, I'm certain" — can be a false positive generated by a flawed cognitive process.

Medical Screening: The Bayes Problem

Medical testing is where false positives have their most direct everyday impact, and where the mathematics of their interpretation is most commonly misunderstood.

Consider a disease that affects 1% of the population. A test for this disease has a 95% sensitivity (true positive rate) and a 95% specificity (true negative rate). A patient tests positive. What is the probability that they actually have the disease?

Most people — including many clinicians — intuitively answer "95%." The correct answer, via Bayes' theorem, is about 16%. Of every 1,000 people tested: 10 actually have the disease, and the test correctly identifies 9.5 of them (sensitivity = 95%). Of the 990 who don't have the disease, the test incorrectly flags 49.5 as positive (1 - specificity = 5%). So of the ~59 positive results, only 9.5 are true positives. That's about 16%.

This is the Base Rate Fallacy in clinical action: the low prevalence of the disease in the tested population means that even a highly accurate test generates far more false positives than true positives when applied to an unselected population. This is why mass screening for rare conditions is so problematic — the mathematics of false positives dominate the results.

Mammography screening for breast cancer has been the subject of extended scientific debate precisely because of this dynamic. A 2013 Cochrane review estimated that for every 2,000 women screened over ten years, one will have her life prolonged, while ten healthy women will be unnecessarily treated (false positives leading to surgery, radiation, or chemotherapy). The false positive rate in US mammography programmes is approximately 50-60% over ten annual screenings — meaning roughly half of women who screen regularly will receive at least one false positive result, with the attendant anxiety, further testing, and sometimes unnecessary treatment.

Prostate-specific antigen (PSA) testing for prostate cancer offers an even starker picture. PSA levels can be elevated by benign conditions, and many slow-growing prostate cancers detected by PSA screening would never have caused symptoms or death during the patient's natural lifetime. The US Preventive Services Task Force for years recommended against routine PSA screening in part because the harms from false positives (unnecessary biopsies, treatment side effects) outweighed the benefits from true positives.

Scientific Research: The Replication Crisis

The false positive problem has become central to the ongoing reckoning in empirical science known as the replication crisis. When researchers across psychology, medicine, nutrition, and other fields attempted to replicate published findings, a startlingly large proportion failed to reproduce.

A landmark 2015 project published in Science — the Reproducibility Project, led by Brian Nosek — attempted to replicate 100 psychology studies published in top journals. Only 36% of the replications produced statistically significant results matching the original. The effect sizes in successful replications were, on average, half those of the originals. Many of the original findings, in other words, were false positives: pattern detections in noise.

The structural cause is that the α = 0.05 threshold, applied to large numbers of studies, guarantees a false positive rate of 5% per test. With thousands of researchers testing thousands of hypotheses, hundreds of false positives will be published — and published findings are selected for because they are positive (publication bias). P-hacking — selectively analysing data until a significant result appears — amplifies this further. The result is a published literature systematically contaminated with false positives that look identical to true ones.

Security Systems: The Alert Fatigue Problem

In cybersecurity, fraud detection, and airport security, false positives create a distinctive operational problem known as alert fatigue. Security systems that generate many alarms cause their operators to become desensitised — when the vast majority of alerts are false positives, operators begin ignoring or rapidly dismissing alerts, including eventually the true ones.

Studies of hospital clinical alarm systems have found that in some intensive care units, staff respond to fewer than 10% of alarms because the false positive rate is so high. A review published in JAMA Internal Medicine found ICU patients could receive over 700 alarms per day, the overwhelming majority clinically irrelevant. The systems designed to prevent harm through vigilance had trained staff to tune them out.

Financial fraud detection faces the same tension. A fraud model tuned to maximise sensitivity will flag many legitimate transactions as fraudulent, generating customer friction, blocked payments, and support costs that can exceed the fraud losses the system was designed to prevent. The optimal operating point involves accepting some false negatives (actual fraud that gets through) to reduce false positives to a manageable rate.

The Alpha-Beta Trade-off

One of the deepest insights in hypothesis testing is that Type I and Type II errors trade off against each other. For a fixed sample size, reducing the false positive rate (setting a lower α) automatically increases the false negative rate (failing to detect real effects). Making the conviction threshold stricter means fewer innocent people convicted, but more guilty people acquitted.

The right balance depends on the context and the relative costs of each type of error. This is not a statistical question — it is an ethical and social question that statistics can inform but not answer. In criminal law, we set the bar high (low α) because we judge false convictions worse than false acquittals. In screening for a highly lethal, curable disease, we may set the bar low (higher α) because a missed case is more catastrophic than a false alarm that prompts further testing.

The error many people make is treating α = 0.05 as a universal standard rather than a context-specific choice. Different domains have different cost structures. A signal intelligence analyst deciding whether to act on a warning of an imminent attack operates in a different error-cost environment than a drug trial statistician testing a new antidepressant. Neither 5% nor any other fixed threshold is universally correct.

Recognising False Positives in Practice

False positives are most likely to mislead when:

• The prior probability (base rate) is low. Testing for rare conditions produces more false positives than true ones, even with accurate tests. Always ask: how common is the thing I'm testing for in this population?
• Multiple hypotheses are tested simultaneously. Testing 20 hypotheses at α = 0.05 gives an expected one false positive by chance alone, even if nothing is real. Corrections (Bonferroni, Benjamini-Hochberg) exist for this but are not always applied.
• The result is surprising or counter-intuitive. Surprising findings have lower prior probabilities of being true, which means a "significant" result is more likely to be a false positive. Extraordinary claims require extraordinary evidence, not just p < 0.05.
• The finding hasn't been replicated. A single study showing a remarkable effect is weak evidence. A finding replicated across multiple independent studies and methods is much more credible.

The companion error — Type II Error / False Negative — is missing something that is really there. Understanding both errors, and the trade-off between them, is the foundation of statistically literate decision-making.

Sources & Further Reading

• Loftus, Elizabeth F., and Katherine Ketcham. Witness for the Defense: The Accused, the Eyewitness, and the Expert Who Puts Memory on Trial. St. Martin's Press, 1991.
• Open Science Collaboration. "Estimating the Reproducibility of Psychological Science." Science 349, no. 6251 (2015): aac4716.
• Gøtzsche, Peter C., and Karsten Juhl Jørgensen. "Screening for Breast Cancer with Mammography." Cochrane Database of Systematic Reviews, 2013.
• Neyman, J., and E.S. Pearson. "On the Problem of the Most Efficient Tests of Statistical Hypotheses." Philosophical Transactions of the Royal Society A 231 (1933): 289–337. (Original formalization of Type I and II errors.)
• Kahneman, Daniel. Thinking, Fast and Slow. Farrar, Straus and Giroux, 2011.
• Wikipedia: Type I and type II errors
• See also: Type II Error / False Negative, P-Hacking, Base Rate Fallacy, Double-Dipping / Circular Analysis