Why Physician Validation Cannot Be Automated

The Automation Bias Problem

Automation bias — the tendency to over-rely on automated decision aids even when they conflict with one's own judgement — is one of the most extensively documented risks in clinical decision support research. A 2022 systematic review in npj Digital Medicine examining clinician interaction with AI-based diagnostic tools found that accuracy on cases where the AI was wrong dropped significantly when clinicians were shown an incorrect AI suggestion compared to when they reviewed the same case unaided.

This is not an argument against AI-assisted clinical workflows. It is an argument for designing them correctly. The distinction that matters is between AI as a decision-support input and AI as a decision-making substitute. The former preserves clinician judgement as the final and accountable step; the latter erodes it.

What the Evidence Shows

Research on clinical decision support systems (CDSS) consistently identifies three failure modes when human oversight is weakened: alert fatigue leading to override of correct warnings, automation complacency leading to acceptance of incorrect outputs, and skill degradation in clinicians who rely on the tool over extended periods. The Institute of Medicine's health IT safety framework explicitly names human-system interaction as one of three core sociotechnical dimensions that must be actively managed, not assumed.

This is the architectural principle behind structured physician validation layers: every AI-generated clinical output is presented as a draft for review, not a finding for acceptance. The clinician sees the underlying evidence, the confidence basis, and the gaps — and makes the final determination. The system is designed so that bypassing physician review is not a configuration option.

Designing for Accountability, Not Just Accuracy

A model can be highly accurate on average and still produce a dangerous individual recommendation. Clinical accountability frameworks — including GMC guidance on AI-assisted practice in the UK and FDA's evolving guidance on Software as a Medical Device — converge on the same requirement: a named, qualified human must remain responsible for the clinical decision, regardless of how the supporting analysis was generated. Removing that layer does not just create legal exposure. It removes the mechanism by which errors are actually caught.