Evidence Traceability in AI-Assisted Clinical Reporting

The Black Box Problem in Clinical AI

A 2023 survey published in JAMA Network Open found that physicians' willingness to act on an AI-generated recommendation was strongly correlated with whether the system disclosed its supporting evidence — not with the system's reported accuracy alone. Clinicians, it turns out, do not simply want a correct answer. They want to know why an answer is correct, because that is what allows them to apply clinical judgement to the specific patient in front of them, who may differ from the population the model was trained on.

This is the core argument for evidence traceability as a design requirement, not a nice-to-have feature. Every clinical recommendation generated by an intelligence system should carry a visible link to the guideline, study, or clinical reasoning step that produced it.

What Regulatory Frameworks Require

The EU's Medical Device Regulation and the FDA's guidance on AI/ML-based Software as a Medical Device both move toward requiring explainability as part of the safety case for clinical AI tools. The EU AI Act classifies most clinical decision-support systems as high-risk, which brings specific transparency and documentation obligations: the system must allow a human reviewer to understand the basis for its output, not merely trust it.

From a medico-legal standpoint, this matters enormously. In a negligence claim, the question is rarely whether a tool was statistically accurate on average — it is whether the standard of care was met in this specific case. A system that can show "this recommendation was derived from NICE guideline NG12, section 1.3.4, applied to a patient meeting these specific criteria" gives the treating physician a defensible, auditable basis for the decision they signed off on.

Trust Is Earned Through Traceability, Not Asserted Through Accuracy

Research on clinician trust calibration — the alignment between how much a clinician trusts a system and how much that trust is actually warranted — consistently shows that opaque high-accuracy systems produce miscalibrated trust: either excessive reliance (when things go well) or premature abandonment (after a single visible error). Transparent systems, even when objectively similar in accuracy, support better-calibrated trust because the clinician can see when a recommendation is well-supported and when it rests on thinner evidence. That distinction is often more clinically useful than the recommendation itself.