Artificial intelligence (AI) in healthcare is rapidly advancing beyond traditional applications. Autonomous AI agents are gaining significant attention for their potential to fundamentally transform medicine. However, researchers at the Else Kröner Fresenius Center (EKFZ) for Digital Health at TUD Dresden University of Technology highlight a growing mismatch between the capabilities of autonomous AI agents and existing medical device regulatory frameworks in the US and Europe. In their publication, they propose innovative regulatory approaches to close this gap while ensuring high patient safety standards. The article has been published in the scientific journal Nature Medicine.

From assistance systems to autonomous agents

The use of large language models (LLMs) and other forms of generative AI (GenAI) in healthcare has surged in recent years, and many of these technologies are already applied in clinical settings. As such, they often qualify as medical devices and must comply with specific laws and regulatory frameworks. Recent decisions have shown that approval is possible for applications with narrowly defined tasks. The next generation of AI systems are broad autonomous AI agents designed to independently carry out complex, goal-directed workflows. They show great potential to support and further improve medical care in the future. AI agents consist of multiple interconnected components, including external databases and computational tools for image analysis, note-taking, clinical guidance, and patient data management. These components are controlled by LLMs that take over decision-making, error handling, and the recognition of completed tasks.

“We are seeing a fundamental shift in how AI tools can be implemented in medicine,” says Jakob N. Kather, Professor of Clinical Artificial Intelligence at the EKFZ for Digital Health at TUD and oncologist at the Dresden University Hospital Dresden. “Unlike earlier systems, AI agents are capable of managing complex clinical workflows autonomously. This opens up great opportunities for medicine – but also raises entirely new questions around safety, accountability, and regulation that we need to address,” he adds.
 

AI agents challenge existing regulatory frameworks

Current medical device regulations were designed for static, narrowly focused technologies that maintain human oversight and do not evolve after entering the market. In contrast, new autonomous and broad-scope technologies have fundamentally different characteristics. They demonstrate greater autonomy, adaptability, and scope than previous AI technologies. As they are capable of autonomously executing complex workflows, they present significant challenges for regulators and developers.

“To facilitate the safe and effective implementation of autonomous AI agents in healthcare, regulatory frameworks must evolve beyond static paradigms. We need adaptive regulatory oversight and flexible alternative approval pathways,” says Oscar Freyer, lead author of the publication and research associate in the team of Professor Stephen Gilbert, who leads the Medical Device Regulatory Science group at EKFZ for Digital Health, TUD.

Rethinking regulation: approval pathways for safe, innovative AI technologies

As AI agents’ capabilities exceed the scope of current regulatory frameworks, the researchers propose several potential solutions to overcome barriers to their implementation.

Immediate adaptations include extending enforcement discretion policies, where regulators acknowledge a product qualifies as a medical device but choose not to enforce certain requirements. Alternatively, a non-medical device classification could be applied to systems that serve a medical purpose but fall outside the traditional medical device regulation.

Medium-term solutions involve developing voluntary alternative pathways (VAPs) and adaptive regulatory frameworks that supplement existing approval processes. These adaptive pathways would shift from static pre-market approval to dynamic oversight using real-world performance data, stakeholder collaboration, and iterative updates. In case of misconduct, devices may be transferred to the established pathways.

The researchers also discuss long-term solutions, such as regulating AI agents in a similar way to the qualification of medical professionals. In this model, regulation would be carried out through structured training processes, in which systems gain autonomy only after demonstrating safe and effective performance.

The researchers note that while tools such as regulatory sandboxes offer some flexibility for testing new technologies, they are not scalable solutions for widespread deployment due to the resources and commitment required by regulatory authorities.

In their article, the researchers argue that meaningful implementation of autonomous AI agents in healthcare will likely remain impossible in the medium term without substantial regulatory reform. VAPs and adaptive pathways are highlighted as the most effective and realistic strategies to achieve this goal. The authors underscore the need for collaborative efforts between regulators, healthcare providers, and technology developers to create frameworks that meet the AI agents' unique characteristics while ensuring patient safety.

“Realizing the full potential of AI agents in healthcare will require bold and forward-thinking reforms,” says Stephen Gilbert, Professor of Medical Device Regulatory Science at the EKFZ for Digital Health at TU Dresden and last author of the paper. “Regulators must start preparing now to ensure patient safety and provide clear requirements to enable safe innovation,” he adds.

Publication:
Oscar Freyer, Sanddhya Jayabalan, Jakob N. Kather, and Stephen Gilbert: Overcoming Regulatory Barriers to the Implementation of AI Agents in Healthcare, Nature Medicine, 2025.
Link: https://www.nature.com/articles/s41591-025-03841-1
DOI: 10.1038/s41591-025-03841-1

Else Kröner Fresenius Center (EKFZ) for Digital Health:
The EKFZ for Digital Health at TU Dresden and University Hospital Carl Gustav Carus Dresden was established in September 2019. It receives funding of around 40 million euros from the Else Kröner Fresenius Foundation for a period of ten years. The center focuses its research activities on innovative, medical and digital technologies at the direct interface with patients. The aim here is to fully exploit the potential of digitalization in medicine to significantly and sustainably improve healthcare, medical research and clinical practice.

Contact:
EKFZ for Digital Health
Anja Stübner and Dr. Viktoria Bosak
Public Relations
Tel.: +49 351 – 458 11379

digitalhealth.tu-dresden.de