🧪 This platform is in early beta. Features may change and you might encounter bugs. We appreciate your patience!
insensitivity_to_sample_size
The tendency to underappreciate the role of sample size in the reliability of statistical results. People expect small samples to be just as representative as large ones, leading them to draw strong conclusions from insufficient data. This violates the law of large numbers, which states that larger samples are more reliable.
A hospital administrator compares two surgical techniques. Technique A succeeded in 8 out of 10 cases (80%) and Technique B in 75 out of 100 cases (75%). The administrator chooses Technique A, ignoring that the small sample of 10 is far less reliable and the 80% could easily be due to chance.
A travel blogger reads that 4 out of 5 guests at a boutique hotel left five-star reviews and enthusiastically recommends it to her followers. She overlooks that only five guests had reviewed the hotel, making the 80% rating far less reliable than a competing hotel's 76% rating from 2,000 reviews.
A nutrition researcher presents early findings showing that 6 out of 7 participants in a pilot study felt more energetic on a new supplement. Headlines declare the supplement 'highly effective,' ignoring that a sample of seven is far too small to draw any meaningful conclusions about the general population.
Binary (yes/no) questions an LLM must answer to identify this aspect:
Are conclusions drawn from a very small number of observations?
Type: binaryIs sample size mentioned or considered in the reasoning?
Type: binaryWould the conclusion be equally confident with awareness of sample limitations?
Type: binaryThe tendency to underappreciate the role of sample size in the reliability of statistical results. People expect small samples to be just as representative as large ones, leading them to draw strong conclusions from insufficient data. This violates the law of large numbers, which states that larger samples are more reliable.
People assess probability using the representativeness heuristic, judging by similarity to the expected outcome. Since both small and large samples can produce the same percentage, people treat them as equally informative.
Always consider sample size when evaluating statistics. Be skeptical of conclusions drawn from small samples and look for larger datasets before making decisions.
This bias affects medical research interpretation, product reviews (giving weight to a few reviews), small-school effects in education policy, and A/B testing in technology companies.
Believing that small samples accurately represent the underlying population distribution.
A study with too few participants or observations to reliably detect the effect being investigated. Low statistical power increases both false negatives and the rate at which significant findings are false positives.
Ignoring general statistical base rates in favor of specific individual-case info.
The mistaken belief that if an event has occurred more frequently than expected in the past, it is less likely to happen in the future (and vice versa), even when events are independent.
The tendency to overestimate the accuracy of one's judgments, especially when available information is internally consistent, even if the information is limited or unreliable.
Use these tools to detect, analyze, or train this aspect.