Oct 9, 2026 · 23 min
Predictable AI Errors Matter More Than Accuracy Alone
Designing for Predictability: How Mental Models of AI Error Boundaries Shape Human-AI Team Performance in High-Stakes Decision Making
High-stakes teams need to know when an AI is likely to fail, because misplaced trust and distrust can make strong systems harmful in practice.
- 1Consistent error patterns help people calibrate when to trust, question, or override an AI system.
- 2Simpler models and manageable task complexity can preserve human understanding without necessarily sacrificing useful performance.
- 3Model updates should be treated as organizational trust events, with safeguards for shared learning and recalibration.
Don't miss
The COMPAS example crystallizes the argument that hundreds of features can make AI errors harder for people to understand and correct.
The brief
A hospital deploys a highly accurate readmission-prediction system, yet the human-AI team fails to improve and sometimes performs worse, exposing the gap between laboratory performance and real-world decisions.
The key distinction is between a model's decision boundary and its error boundary: users need a mental model of specific blind spots, not just an overall accuracy score.
Parsimony, consistent failure patterns, and manageable task complexity make errors easier to recognize; random failures and hundreds of opaque features overwhelm human pattern recognition.
Software updates can erase a user's learned expectations even when accuracy improves, so organizations should treat model changes as trust events with training and failure-case libraries.
The episode closes on an unresolved ethical choice: when should a slightly less accurate but more predictable model take priority over raw performance?