AI in Healthcare: Regulation Is Coming. Are You Ready?

Most Clinical AI Tools Are Now "High-Risk" by Default

The EU Artificial Intelligence Act, which entered into force in August 2024 with phased compliance deadlines extending into 2027, classifies AI systems used as safety components of medical devices — or as standalone software intended for diagnosis, treatment recommendation, or triage — as high-risk by default under Annex III. This is not a niche classification: it captures the large majority of AI-based clinical decision support tools currently in commercial use, including triage assistants, diagnostic aids, and treatment pathway recommendation systems.

High-risk classification triggers a specific set of obligations: a documented risk management system maintained across the product lifecycle, technical documentation sufficient for a regulator to assess conformity, human oversight measures that are demonstrable rather than asserted, and post-market monitoring with incident reporting.

The Overlap — and Gap — With Medical Device Regulation

Many clinical AI tools already fall under the EU Medical Device Regulation (MDR) if they meet the definition of Software as a Medical Device. The AI Act does not replace MDR obligations — it layers additional requirements on top, with the European Commission's guidance indicating that conformity assessment processes will increasingly need to address both frameworks jointly. Providers who have only mapped their MDR compliance, without separately assessing AI Act high-risk obligations, are working from an incomplete compliance picture, particularly regarding the AI Act's explicit human oversight and technical documentation requirements, which are more prescriptive than MDR's general safety and performance requirements.

What "Demonstrable Human Oversight" Actually Requires

Article 14 of the AI Act requires that high-risk AI systems be designed so that natural persons can effectively oversee their operation, including the ability to understand the system's capabilities and limitations, to correctly interpret its output, and to decide not to use it or to override an output. Critically, regulatory guidance has emphasised that human oversight cannot be satisfied by a theoretical ability to override — the system's interface and workflow must make override genuinely practical, with sufficient information presented at the point of decision. A system that buries the override option, or presents AI output with such authority that override becomes socially or practically difficult, does not meet the bar even if override is technically possible.

For providers evaluating or building clinical AI tools, the practical preparation step is straightforward to state and hard to retrofit: human oversight, evidence traceability, and documented limitations need to be architectural decisions made before deployment, not compliance documentation written after the fact to describe a system that was not designed with them in mind.