🧪 This platform is in early beta. Features may change and you might encounter bugs. We appreciate your patience!
automation_bias
Automation bias is the tendency to favor suggestions from automated systems and to ignore or discount contradictory information from non-automated sources, even when the automated system is demonstrably wrong. As systems become more sophisticated, people increasingly defer to their outputs without applying critical evaluation, effectively abdicating judgment to the machine.
A pilot follows GPS navigation instructions to taxi onto a closed runway despite clearly visible physical barriers and warning signs, because the automated system indicated that route was correct.
A radiologist reviewing AI-assisted scans passes over a suspicious area that her own eye flagged because the algorithm did not highlight it, later discovering it was an early-stage tumor the software had missed — her trust in the system overrode her own clinical judgment.
A warehouse manager ignores repeated warnings from floor staff that the automated inventory system is double-counting returned items, insisting the software 'would flag it if there were a real problem,' until a year-end audit reveals a significant stock discrepancy.
Binary (yes/no) questions an LLM must answer to identify this aspect:
Is an automated recommendation accepted without critical evaluation?
Type: binaryAre human observations or contradictory evidence dismissed in favor of system output?
Type: binaryWould the same conclusion be reached without the automated suggestion?
Type: binaryAutomation bias is the tendency to favor suggestions from automated systems and to ignore or discount contradictory information from non-automated sources, even when the automated system is demonstrably wrong. As systems become more sophisticated, people increasingly defer to their outputs without applying critical evaluation, effectively abdicating judgment to the machine.
Automated systems are perceived as objective and infallible, triggering authority bias. Monitoring an automated system for errors is cognitively demanding and boring, leading to vigilance decrement. The effort of verifying automated output often exceeds the effort of simply accepting it.
Maintain situational awareness independently of automated systems and treat their outputs as recommendations to be verified rather than instructions to be followed. Regularly practice manual decision-making to maintain skills.
Automation bias has contributed to aviation accidents, medical errors from electronic health records and decision support systems, and navigation disasters where drivers follow GPS into lakes or off cliffs.
Use these tools to detect, analyze, or train this aspect.