🧪 This platform is in early beta. Features may change and you might encounter bugs. We appreciate your patience!
confidence_interval_misinterpretation
A 95% confidence interval is widely misinterpreted as meaning there is a 95% probability that the true parameter lies within the calculated interval. The correct interpretation is procedural: if the study were repeated infinitely many times, 95% of the intervals constructed would contain the true parameter. For any specific interval, the true parameter is either inside or outside — a statement about procedure, not about any particular interval.
A study reports that the treatment effect is 5.2 points (95% CI: 2.1, 8.3). A journalist writes 'there is a 95% chance the true effect is between 2.1 and 8.3 points.' This is incorrect — the 95% refers to the long-run performance of the procedure, not to this specific interval.
A polling firm reports that a candidate's approval rating is 52% (95% CI: 49%, 55%). A news anchor tells viewers: 'We are 95% confident the candidate's true approval is somewhere in that range right now.' In fact, the interval is a property of the repeated sampling procedure — the true approval is a fixed value, and this particular interval either captures it or it doesn't.
A pharmaceutical company's press release states: 'There is a 95% probability that our drug reduces blood pressure by between 3 and 9 mmHg.' A statistician flags this as misleading — the 95% refers to the long-run frequency with which such intervals, constructed from repeated studies, would contain the true fixed effect, not to a probability about this specific interval.
Binary (yes/no) questions an LLM must answer to identify this aspect:
Is a 95% CI being described as a range that contains the true parameter with 95% probability?
Type: binaryIs the CI being interpreted as a Bayesian credible interval rather than a frequentist confidence interval?
Type: binaryIs the CI being used to infer that all values outside the interval are equally unlikely?
Type: binaryDoes the claim treat the width of the CI as directly measuring uncertainty about a specific interval?
Type: binaryA 95% confidence interval is widely misinterpreted as meaning there is a 95% probability that the true parameter lies within the calculated interval. The correct interpretation is procedural: if the study were repeated infinitely many times, 95% of the intervals constructed would contain the true parameter. For any specific interval, the true parameter is either inside or outside — a statement about procedure, not about any particular interval.
The correct interpretation is procedural and abstract; the incorrect interpretation is intuitive and direct. Bayesian credible intervals do have the probabilistic interpretation, so the conceptual confusion is easy to make.
State CIs in procedural terms. Use Bayesian credible intervals when a probabilistic statement about a specific interval is required. Do not conflate CI width with posterior uncertainty.
Surveys of scientists find that over 80% misinterpret confidence intervals, affecting peer review, public communication, and policy decisions.
Use these tools to detect, analyze, or train this aspect.