🧪 This platform is in early beta. Features may change and you might encounter bugs. We appreciate your patience!
law_of_small_numbers
The law of small numbers is the erroneous belief that small samples should be representative of the population from which they are drawn, mirroring the statistical properties of the population in miniature. Named as an ironic counterpart to the actual law of large numbers, it reflects the cognitive tendency to expect patterns and regularities even in sequences too short to reliably display them. This leads to premature generalization, overinterpretation of noise, and false confidence in unreliable data.
A school district observes that three small rural schools (each with 30 students) rank among the top 10 in state test scores and concludes small schools are superior. They fail to notice that three other small schools rank in the bottom 10. Small schools appear at both extremes because their small samples produce volatile averages — not because of school quality.
An investor notices that a particular stock-picking newsletter correctly predicted the market direction three months in a row and immediately moves his savings into the recommended portfolio, convinced the analyst has a genuine edge — ignoring that with hundreds of newsletters, a few will get three in a row purely by chance.
A restaurant owner tries a new social media ad campaign for two weekends and gets unusually high foot traffic both times. She immediately cancels all other marketing and doubles her ad budget, not realizing that two weekends is far too small a sample to distinguish a real effect from normal weekly variation.
Binary (yes/no) questions an LLM must answer to identify this aspect:
Is a small sample being treated as if it accurately represents the population?
Type: binaryAre patterns observed in a small sample being assumed to be stable and generalizable?
Type: binaryHas the analysis failed to consider that small-sample results may simply reflect random variation?
Type: binaryWould the conclusion change substantially if based on a much larger sample?
Type: binaryThe law of small numbers is the erroneous belief that small samples should be representative of the population from which they are drawn, mirroring the statistical properties of the population in miniature. Named as an ironic counterpart to the actual law of large numbers, it reflects the cognitive tendency to expect patterns and regularities even in sequences too short to reliably display them. This leads to premature generalization, overinterpretation of noise, and false confidence in unreliable data.
The human mind is designed to extract patterns quickly, which was adaptive in our evolutionary environment but leads us astray with statistical data. We intuitively apply a mental version of the law of large numbers to samples of any size, expecting even tiny samples to mirror the population faithfully.
Recognize that small samples naturally produce extreme and variable results. Demand larger samples before drawing conclusions. Use formal statistical tests that account for sample size. Be especially suspicious of impressive-looking results from very small datasets.
Affects medical decisions (rare case reports driving treatment choices), business strategy (pivoting based on a few customer interactions), and sports (judging player ability from a handful of games).
The tendency to draw strong conclusions from small samples, failing to recognize that small samples are more variable and less reliable than large ones.
The mistaken belief that if an event has occurred more frequently than expected in the past, it is less likely to happen in the future (and vice versa), even when events are independent.
Ignoring general statistical base rates in favor of specific individual-case info.
A study with too few participants or observations to reliably detect the effect being investigated. Low statistical power increases both false negatives and the rate at which significant findings are false positives.
Attributing natural fluctuation to a specific intervention.
The tendency to overestimate the accuracy of one's judgments, especially when available information is internally consistent, even if the information is limited or unreliable.
Use these tools to detect, analyze, or train this aspect.