🧪 This platform is in early beta. Features may change and you might encounter bugs. We appreciate your patience!
illusion_of_validity
The tendency to maintain confidence in predictions and judgments even when the evidence shows they are unreliable. People continue to trust their intuitive assessments despite statistical feedback showing poor accuracy. The subjective experience of 'seeing' a pattern maintains confidence even when that pattern has no predictive value.
An interviewer is highly confident in their ability to predict job performance after a 30-minute unstructured interview, despite extensive research showing that unstructured interviews have very low predictive validity compared to structured assessments and work samples.
A venture capitalist insists they can identify winning founders within the first ten minutes of a pitch meeting based on their 'gut feeling' about the founder's passion and presence, despite their own portfolio showing no correlation between their initial enthusiasm and actual startup success.
An experienced wine sommelier is utterly confident rating wines by taste alone, certain they can distinguish expensive vintages from cheap ones. When tested blind, their accuracy falls to near chance levels — yet their subjective certainty remains completely unshaken by the results.
Binary (yes/no) questions an LLM must answer to identify this aspect:
Is confidence boosted by consistency of information rather than its quality?
Type: binaryIs the information internally coherent but potentially incomplete or unreliable?
Type: binaryWould confidence decrease if the limitations of the data were made explicit?
Type: binaryThe tendency to maintain confidence in predictions and judgments even when the evidence shows they are unreliable. People continue to trust their intuitive assessments despite statistical feedback showing poor accuracy. The subjective experience of 'seeing' a pattern maintains confidence even when that pattern has no predictive value.
The coherence of a narrative or pattern feels convincing regardless of its actual predictive power. When information fits together into a compelling story, the subjective experience of understanding generates confidence that is not calibrated to actual accuracy.
Track your prediction accuracy over time and compare it to base rates and statistical models. Replace unstructured intuitive judgment with structured, evidence-based assessment tools wherever possible.
The illusion of validity affects hiring decisions, clinical diagnosis, financial forecasting, and any domain where experts make predictions based on pattern recognition. It is why actuarial methods often outperform expert judgment.
The tendency to draw strong conclusions from small samples, failing to recognize that small samples are more variable and less reliable than large ones.
Believing that small samples accurately represent the underlying population distribution.
A model or analysis fits the noise in the training data so closely that it fails to generalize to new data. The model captures random fluctuations rather than the underlying pattern.
Ignoring general statistical base rates in favor of specific individual-case info.
Attributing natural fluctuation to a specific intervention.
Use these tools to detect, analyze, or train this aspect.