Large Language Models in Medicine: New Review Analyzes Risks and Strategies for Safe Use in Clinical Practice

Large language models (LLMs), including the models behind ChatGPT and Claude as well as numerous other systems developed specifically for medical applications, are increasingly used in clinical workflows. They support medical documentation, summarize knowledge, and assist with clinical decision-making. However, their adoption is outpacing the development of systems for oversight and safety. An interdisciplinary team of researchers at the Else Kröner Fresenius Center (EKFZ) for Digital Health at TUD Dresden University of Technology and University Hospital Dresden, together with national and international colleagues, has systematically analyzed the risks associated with LLM use in medicine. The review, published in Nature, brings together evidence from medical AI, cybersecurity, regulatory science, ethics and behavioral psychology and outlines strategies for trustworthy and responsible use of artificial intelligence (AI) in clinical practice.

LLMs have the potential to support and enhance the work of healthcare professionals in a variety of areas. These tools are already being used in practice, often without institutional guidance or clear rules. This creates new demands and a need for action regarding patient safety, data protection, and accountability. The authors of the newly published review show that these risks can arise throughout the entire lifecycle of AI systems: from initial model design to training data, model deployment, and real-world use in clinical environments. They distinguish different types of risks:

Security risks, which can arise from the manipulation of training data (“data poisoning”), targeted interference with model behavior, or so-called prompt injections. The latter refers to hidden instructions inserted in user prompts leading to wrong or even dangerous outcomes, e.g. failing to detect a tumor in a tissue sample despite it being visible. Additionally, weaknesses in the IT infrastructure can expose sensitive patient data or disrupt systems.

Model-inherent safety risks: LLMs can generate seemingly plausible yet incorrect information, known as “hallucinations.” This is particularly critical in clinical settings, because incorrect diagnoses or recommendations can put patient safety at risk. The models may also adapt their responses too strongly to user expectations, thereby reinforcing incorrect assumptions.

Human-AI interaction risks: The way clinicians interact with these systems can influence clinical decisions. Confident or persuasive responses may lead to overreliance (automation bias) or reinforce existing beliefs (confirmation bias). Complex or lengthy interactions can further reduce the reliability of the answers.

The review also highlights structural and ethical challenges. Many systems are not locally hosted, raising questions about data control and privacy. Furthermore, the informal use of LLMs, referred to as shadow use, is already occurring in clinical settings, frequently without any official safeguards.

“Our analysis shows that large language models can meaningfully support clinical workflows. Their safe use, however, cannot be taken for granted. Risks arise at many stages and must be addressed systematically and comprehensively before and alongside clinical implementation,” says Dr. Jan Clusmann, postdoctoral researcher in the group of Professor Jakob N. Kather at EKFZ for Digital Health at TUD and first author of the publication.

Strategies for safer implementation

To reduce risks, the authors propose several measures, including secure development processes, careful curation of training data, systematic evaluation, and continuous monitoring of models, as well as clear responsibilities within healthcare institutions. They emphasize that safety is not only a technical issue but requires coordinated efforts across research, clinical practice, and regulation. Human oversight remains essential, the authors emphasize.

“The development of AI for healthcare does not end with building powerful models. It is equally important to rigorously evaluate their safety, transparency, and value in clinical practice. By providing an evidence-based foundation for this work, our researchers are making an important contribution to the responsible digital transformation of medicine,” says Prof. Esther Troost, Dean of the Carl Gustav Carus Faculty of Medicine at TU Dresden.

Specifically, the researchers recommend establishing clear structures, such as local teams dedicated to overseeing the use of AI systems in clinical practice. In addition, they also recommend setting up centralized units – so-called Security Operations Centers (SOCs) for AI – to detect incidents across institutions and enable coordinated responses.

“AI is already being used in healthcare, often without formal oversight. The key question is how to implement these systems in a way that is transparent, robust, and aligned with clinical responsibility,” says Prof. Jakob N. Kather, Professor of Clinical AI at EKFZ for Digital Health at TUD, physician at University Hospital Dresden and researcher at National Center for Tumor Diseases (NCT) Heidelberg.

Adapting regulation to dynamic AI systems

The review also highlights gaps in current regulation. Only a small proportion of AI systems are formally approved as medical devices. Current regulatory frameworks were not designed for adaptive technologies that continuously evolve, such as AI-based software.

“To ensure both patient safety, and timely patient and health system benefit from such AI systems we need suited regulatory approaches that provide consistent evaluation and continuous monitoring. Technological development and oversight and surveillance approaches must be more closely integrated to safely utilize the potential of large language models,” says Prof. Stephen Gilbert, Professor of Medical Device Regulatory Science at EKFZ for Digital Health at TUD.

This review article is a joint effort at the following national and international institutions:

Else Kröner Fresenius Center (EKFZ) for Digital Health, Faculty of Medicine and University Hospital Carl Gustav Carus, TUD Dresden University of Technology, University Hospital RWTH Aachen, German Cancer Research Center (DKFZ) Heidelberg, National Center for Tumor Diseases (NCT) Heidelberg, University Hospital Heidelberg, Faculty of Medicine at Heidelberg University, Faculty of Medicine Mannheim, University Medical Center Mainz, Purdue University West Lafayette, University of Pennsylvania, and the patient right network The Light Collective Eugene.

Publication

Clusmann J, Freyer O, Ostermann M, Ferber D, Ghaffari Laleh N, Hilgers L, Kolbinger FR, Schneider CV, Downing A, Wekenborg MK, Gilbert S, Foersch S, Truhn D, Wiest IC, Kather JN. Safety and security of large language models in healthcare. Nature, 2026.

Link: https://www.nature.com/articles/s41586-026-10687-1

Else Kröner Fresenius Center (EKFZ) for Digital Health

The EKFZ for Digital Health at the Faculty of Medicine at TUD Dresden University of Technology and University Hospital Carl Gustav Carus Dresden was established in September 2019. It receives funding of around 40 million euros from the Else Kröner Fresenius Foundation for a period of ten years. The center focuses its research activities on innovative, medical and digital technologies at the direct interface with patients. The aim here is to fully exploit the potential of digitalization in medicine to significantly and sustainably improve healthcare, medical research and clinical practice.

Contact

EKFZ for Digital Health
TUD Dresden University of Technology
Anja Stübner and Dr. Viktoria Bosak
Science Communication
Tel.: +49 351 – 458 11379

digitalhealth.tu-dresden.de