Safe use of large language models in medicine – creating and complying with clear regulatory requirements

In the journal “The Lancet Digital Health”, researchers led by Professors Stephen Gilbert and Jakob N. Kather from the Else Kröner Fresenius Center for Digital Health at the TUD Dresden University of Technology provide an overview of current health applications based on large language models (LLMs), their strengths and weaknesses and the regulatory challenges. They call for a framework that recognizes the capabilities and limitations of these AI applications and emphasize the urgent need to enforce existing regulations. A continued hesitant approach on part of authorities not only endangers users, but also the potential of future LLM applications in medicine.

Opportunities and risks of LLM-based healthcare applications
Applications based on artificial intelligence (AI) are already indispensable in medicine. LLMs such as GPT-4 offer a wide range of possibilities for supporting the diagnosis, treatment and care of patients. At the same time, there are concerns regarding their safety and regulation. Results are often variable, lack explainability and carry the risk of hallucinations. The approval of LLM-based applications for medical purposes as medical devices under US and EU law poses challenges for regulatory authorities. Despite the risks for users, such as misdiagnosis or unverified medical advice, various applications are already available on the market.

Since the introduction of LLMs, developers such as Google, Meta, OpenAI and Microsoft have continuously introduced new and improved models. Their performance in medical applications has also improved. “LLMs have the potential to transform healthcare and will become increasingly important in the areas of cancer diagnosis and treatment, as well as screening, remote care, documentation and clinical decision support. These models offer great potential, but also harbor risks,” says Prof. Dr. med. Jakob N. Kather, Professor of Clinical Artificial Intelligence at the EKFZ for Digital Health at TU Dresden and oncologist at the University Hospital Carl Gustav Carus in Dresden. Intensive research is still being carried out to determine whether the advantages outweigh the disadvantages in medical applications. In addition to the many possibilities, the researchers clearly point out open legal and ethical questions in their publication, particularly with regard to data privacy, the protection of intellectual property and the problem of gender-specific and racial biases.

Approval as medical devices required
As soon as applications offer specific medical advice on the diagnosis or treatment to laypersons, they are medical devices under both EU and US law and as such require regulatory approval. While compliance with these regulations was relatively clear for previous, narrowly defined applications, the versatility of LLMs poses regulatory challenges for the authorities. Although there are some legal ambiguities, there are also clear cases of unapproved unregulated products being on the market. The researchers make it clear that the authorities have the responsibility to enforce the applicable rules. At the same time, they should ensure that appropriate frameworks for testing and regulating health applications based on LLMs are developed. “These technologies are already with us, and we need to have two things together to ensure their safe development.  The first are appropriate methods to evaluate these new technologies. The second is the appropriate enforcement of current legal requirements against clear cases where unsafe LLM apps are on the market. This is essential if we want to use these medical AI applications safely in the future,” says Prof. Dr. Stephen Gilbert, Professor of Medical Device Regulatory Science at the EKFZ for Digital Health at TU Dresden.

Publication
Oscar Freyer, Isabella Catharina Wiest, Jakob Nikolas Kather, Stephen Gilbert: A future role for health applications of large language models depends on regulators enforcing safety standards; The Lancet Digital Health, 2024.  https://www.thelancet.com/journals/landig/article/PIIS2589-7500(24)00124-9/fulltext

Else Kröner Fresenius Center (EKFZ) for Digital Health
The EKFZ for Digital Health at TU Dresden and University Hospital Carl Gustav Carus Dresden was established in September 2019. It receives funding of around 40 million euros from the Else Kröner Fresenius Foundation for a period of ten years. The center focuses its research activities on innovative, medical and digital technologies at the direct interface with patients. The aim here is to fully exploit the potential of digitalization in medicine to significantly and sustainably improve healthcare, medical research and clinical practice.

Contact
EKFZ for Digital Health
Anja Stübner and Dr. Viktoria Bosak
Public Relations
Tel.: +49 351 – 458 11379

https://digitalhealth.tu-dresden.de/