Reference Class Problem

Also Known As: Reference Class Selection Bias Base Rate Ambiguity

Statistical Error ID: reference_class_problem

Definition

The Reference Class Problem occurs when assigning a probability to an individual case that belongs to multiple groups (reference classes), each yielding a different probability. The choice of reference class can dramatically change the estimated probability, yet there is often no objectively 'correct' class to choose. This problem is pervasive in everyday reasoning, legal contexts, medical diagnoses, and risk assessment.

Examples

A doctor tells a patient with cancer that their condition has a 30% five-year survival rate. But that statistic covers all patients of all ages and health levels. Narrowed to patients of similar age, fitness, and treatment protocol, the rate might be 60% — a very different prognosis, and a very different conversation.

A city planner estimates the cost of a new subway line at €2 billion based on similar projects nationwide. But the narrowest defensible reference class — urban underground extensions in cities with the same geology and labour costs — averages €4 billion. The broad reference class gave a false sense of affordability.

An investor asks: 'What is the probability this startup succeeds?' Depending on the reference class — all startups (10%), all funded startups (20%), all funded SaaS startups with repeat founders (35%) — the answer changes dramatically. Each class is valid; the choice determines the conclusion.

Verification Steps

Verification Steps

Binary yes/no questions that an AI must answer to detect a reasoning pattern in a text.

Each of the 452 aspects has verification steps — simple yes/no questions designed to systematically detect whether a pattern appears in a text. For ad hominem: "Does the argument attack a person rather than their claim?" For false dichotomy: "Are only two options presented when more exist?" This ensures consistent, reproducible analysis.

View in glossary →

Binary (yes/no) questions an LLM must answer to identify this aspect: