🧪 This platform is in early beta. Features may change and you might encounter bugs. We appreciate your patience!
simpsons_paradox
Simpson's Paradox occurs when a trend that appears in several different groups of data reverses or disappears when these groups are combined. This happens because of a lurking variable that changes the composition of the groups. The paradox reveals that aggregated data can tell a fundamentally different story than disaggregated data, making the choice of how to partition data a critical analytical decision.
University admission data shows that overall, women are admitted at a lower rate than men (apparent gender bias). But when broken down by department, women are admitted at equal or higher rates in every single department. The reversal occurs because women disproportionately applied to highly competitive departments with low admission rates for everyone.
An online platform reports that its new recommendation algorithm increases average time spent per user overall. But when broken down by user type, both casual users and power users actually spend less time — the overall increase is driven entirely by a surge in new user sign-ups, who naturally spend more time exploring the platform.
A school district reports that its overall average test scores improved after a new curriculum was introduced. However, when scores are broken down by school, every individual school shows a decline. The district-level improvement is an artifact of higher-performing schools growing in enrollment while lower-performing schools shrank.
Binary (yes/no) questions an LLM must answer to identify this aspect:
Is aggregated data being used to draw a conclusion?
Type: binaryDoes the trend reverse or disappear when the data is broken into subgroups?
Type: binaryIs a confounding variable (lurking variable) driving the reversal?
Type: binarySimpson's Paradox occurs when a trend that appears in several different groups of data reverses or disappears when these groups are combined. This happens because of a lurking variable that changes the composition of the groups. The paradox reveals that aggregated data can tell a fundamentally different story than disaggregated data, making the choice of how to partition data a critical analytical decision.
People trust aggregate statistics as objective summaries. The paradox exploits the assumption that what holds for the whole must hold for the parts, and vice versa.
Always examine data at multiple levels of aggregation. Ask whether the composition of subgroups differs significantly and whether a confounding variable might reverse the observed trend.
Simpson's Paradox famously appeared in the UC Berkeley gender bias case (1973) and regularly surfaces in medical treatment comparisons where patient severity varies between treatment groups.
Drawing broad conclusions from limited, unrepresentative, or anecdotal evidence.
Treatment groups differ in baseline risk, confounding the treatment effect.
Excluding a relevant confounding variable from a model biases the estimated effects.
Statistical results change depending on how geographic boundaries are drawn or aggregated.
The presumed effect is actually the cause, reversing the true causal direction.
Random observation of a process is more likely to catch long-duration events than short ones.
Two individually losing strategies can combine to produce a winning strategy.
A group decides on a course of action that no individual member actually wants.
Use these tools to detect, analyze, or train this aspect.