Dummy des Fraunhofer IGD. Über die Brille sieht der Arzt in Augmented Reality die exakte Lage des Lymphknoten.
© Fraunhofer / Peter Granser

Artificial Intelligence in Medicine

Fraunhofer IGD dummy. The glasses enable the doctor to see, in augmented reality, the exact position of the lymph node.

How a digital twin can save lives

Diagnostic hypotheses and personalized therapies: artificial intelligence has begun to revolutionize the fight against cancer.


It’s considered the epidemic of our time: 18 million people will be stricken with cancer this year. Statistics show that Germany alone sees 500,000 new cases every year. Each year, around 200,000 Germans die of cancer. The World Health Organization’s (WHO) International Agency for Research on Cancer (IARC) calculated that nearly one in four cancer diagnoses worldwide concerns a European – as does one in five tumor-related deaths.

Artificial intelligence is considered the hope of our time. Although AI scares some people, it is now the case that artificial intelligence is assisting humans more and more, especially in medicine. Artificial intelligence in the medical field isn’t science fiction. Based on his experience, Prof. Jörn Kohlhammer, department head at the Fraunhofer Institute for Computer Graphics Research IGD, is quite certain: “AI can benefit physicians by comparing the course of a disease in a large number of patients and offering the doctor suggestions as to the best possible personalized treatment.”

This year, Germany launched a National Decade against Cancer. Kicking off the initiative, Anja Karliczek, German Federal Minister of Education and Research (BMBF), said, “Research is the most important tool in the fight against cancer.” In May, she added, “Over the coming ten years, we will pool all our strengths in the National Decade against Cancer.”

Fraunhofer Lighthouse project MED²ICIN

Digital patient model as a basis for personalized and cost-optimized treatment.

Leitprojekt MED²ICIN
© Fraunhofer IGD

Led by Fraunhofer IGD, researchers from seven Fraunhofer Institutes are now participating in the MED2ICIN lighthouse project with the aim of predicting how well any given treatment will work on any given patient. One person may respond very well to one treatment, while another doesn’t respond at all. When this happens, not only does it burden the patient both physically and emotionally, it also incurs unnecessary costs for health insurance companies.

“To make this prediction, we use a digital patient model,” explains Dr. Stefan Wesarg, a department head at Fraunhofer IGD. “It summarizes all the data available on a patient – data acquired from tests and data relating to prior illnesses – as well as lifestyle information, such as whether the patient smokes. The costs incurred and data on general healthcare costs are also fed into the model.” In other words, one of the aims of the project is to create a digital twin of the patient.

What interests physicians attending to a patient is whether there are similar therapeutic cases and what can be learned from them in order to improve the current treatment. To this end, they classify groups of people with similar clinical pictures and disease progression into cohorts.

However, examining these cohorts for significant similarities or differences is extremely time-consuming and therefore currently impractical. This is precisely where artificial intelligence comes into play: it scours the volumes of data for important matches, visualizes them and determines which treatments are promising for a particular patient. “In this way, doctors can also compare patients they would never meet in person – because, for instance, some diseases are extremely rare,” Kohlhammer adds. In the long run, this tool could help predict which form of treatment would be best for treating a patient’s specific illness.

If it were up to the researchers, these findings would then also feed into the guidelines that prescribe which diagnostic methods and treatment options are to be used for various indications – for instance, which form of chemotherapy should be used for colorectal cancer, or whether and how chemotherapy, radiation and surgery can be combined. “We need to start with the guidelines,” says Kohlhammer, “because they are what provide the recommendations. By using the data from our system, we can reinforce the expert knowledge of doctors and optimize the guidelines in order to provide a more personalized treatment.” For patients, this would mean that they would no longer be treated according to general standards, but rather with the treatment method that promises the best prospects for them personally.

Sending artificial intelligence to work in the “text mines”

Colon cancer continues to be treated largely on the basis of these guidelines. In some patients, however, tumors continue to spread in the body. Particularly in these cases, alternative treatment methods that are not yet included in the guidelines may be of interest. For the doctors treating these patients, this would mean scouring scientific publications on clinical studies in order to track down more-expedient therapies.

This is a complex undertaking – and another area where artificial intelligence can provide targeted support. In the BMBF-funded Electronic Patient Path (EPP) research project, Jil Sander, business unit manager at the Fraunhofer Institute for Intelligent Analysis and Information Systems IAIS, is part of a team working to create a suitable text-mining solution: “AI searches public abstracts of medical publications for therapies whose effectiveness for patient groups can be estimated on the basis of certain biomarkers. The relevant publications are then recommended to the doctor as reading material for potential treatments for colorectal cancer.” A biomarker might be a gene that has mutated in the tumor cells of some, but not all, patients. And it may be that chemotherapy works well for the group with the mutated gene but has hardly any effect in the second group. In this way, doctors can form specific patient groups and search for the best treatment options for a particular group.

But the program doesn’t just search for keywords – a simple search algorithm could do that. Instead, it learns to use context in order to identify certain classifiers, known as entities, and their relationships. This makes it possible, for instance, to identify whether a certain therapy has actually been used for a clinical picture – rather than merely listing the therapies and clinical pictures within a text. The system can also recognize entirely new entities, such as novel treatment options that have not yet been recorded in any database and thus have not yet been keyworded. But the project encompasses more than just these text-mining solutions. Its goal is to develop an Electronic Patient Path (EPP) – an integrated system for therapy guidance for colorectal cancer that extends beyond the guidelines. This is why the scientific partners – Ruhr-Universität Bochum, University of Bonn, Hamm-Lippstadt University of Applied Sciences and Fraunhofer IAIS – are collaborating with a clinical partner, the Universität zu Lübeck and the University Medical Center Schleswig-Holstein, Lübeck campus. The final results are still out. In the long term, however, it is hoped that the program goals will help doctors pinpoint the most relevant new studies and thereby drive up the treatment success rate for cancers – by, for example, continuously refining the EPP method in the MED2ICIN lighthouse project, both for colorectal cancer and for other clinical pictures. But here, too, the artificial intelligence behind the integrated system is meant to support doctors, not replace them.

Dummy des Fraunhofer IGD. Über die Brille sieht der Arzt in Augmented Reality die exakte Lage des Lymphknoten.
© Fraunhofer / Peter Granser

The head and neck region is particularly challenging

Now, however, artificial intelligence can also automatically detect the tumor. As Wesarg explains, “Our software tool localizes and labels the tumor in the computer tomography images, presents it in 3D and analyzes the corresponding image data.” The system is based on neural networks and was trained with data in which the tumor was labeled manually. It then used this data to generate a model. Additional data is added from the head and neck atlas, such as the information that the larynx is completely healthy-looking, so the system doesn’t need to look for the tumor there. The results of the head and neck atlas thereby provide a preselection.

How is brightness distributed within the tumor? Is there anything that isn’t noticeable when a human looks at it? The tool uses various descriptive parameters to answer these questions. In total, with the appropriate software, more than a hundred parameters of this kind can be extracted from the images of a head and neck tumor.

Faster, cheaper and gentler than a biopsy

Initial results show that, with this approach, CT images can even provide information that once could only be obtained through a surgical procedure followed by laboratory analysis of the extracted tumor tissue. “So it’s conceivable, for instance, that a correlation could be found between the intensity pattern within the tumor region and a cell abnormality detected in the lab. With enough patients, it could one day be possible to infer – with statistical certainty – pathological cell changes on the basis of the appearance of the tumor in the image data.” Thus, so the theory goes, it will soon be possible to use artificial intelligence to draw conclusions regarding tissue markers, obviating the need for an actual biopsy. This is easier not only on patients but also on health insurance companies’ budgets. On top of that, the results are available much more quickly than they would be for a biopsy with a lab analysis of the extracted tissue.

Parts of this technology are already being used in initial test runs at the HNO clinic of the University Hospital of Düsseldorf. The doctors there are using the technology to retroactively analyze patient data and review cohort assignments. In the months ahead, this test is expected to reveal how the AI findings correlate with empirical knowledge, thus marking the first step toward cohorting – and onward toward treatment tailored to the individual patient.

The long-term aim is to personalize medical care – to identify the therapy with the highest probability of success for each patient. To achieve this, the algorithms developed for the head and neck region could also be extended to other types of cancer. For this, however, the algorithm needs to have the relevant information as to which structures it should look for in the image data. This is because tumors in the head and neck region have different markers than, say, lung tumors.

In collaboration with MedCom GmbH in Darmstadt, a Fraunhofer IGD spin-off, the researchers also want to begin this process as early as the initial diagnosis. In the BMBF ECHOMICS project, they are using artificial intelligence to analyze ultrasound images of the lymph nodes in a process analogous to a biopsy. This is because a permanent enlargement of the lymph nodes may indicate the presence of a tumor in the body. This would enable doctors to detect tumors much sooner than is currently possible, thus facilitating swifter treatment and improving the chances of success.

Analyzing the “inherited” immune system

When radiation or chemotherapy is not successful in treating leukemia or lymphoma, there is usually only one chance for recovery: a bone marrow or blood stem cell transplant. But here, too, the chances of success are very small. Currently, the majority of patients die despite the transplant. This is because chemotherapy kills all blood stem cells – and thus also all the white blood cells that make up our immune system. In a transplant, the patient therefore “inherits” the donor’s immune system. Yet this may turn against the patient. And it is also possible that the new immune system doesn’t recognize latent pathogens in the patient’s body, so that the corresponding disease erupts again.

The researchers working on the XplOit project at the Fraunhofer Institute for Biomedical Engineering IBMT now aim to use artificial intelligence to predict whether the new immune system will give rise to these kinds of problems. Will the patient tolerate the transplant? What is the probability that they will survive? And, if they do, what is the probability that the cancer will spread again? “Our XplOit platform enables earlier detection and treatment of life-threatening complications than is currently possible,” says project coordinator Stephan Kiefer from Fraunhofer IBMT.

To teach the neural networks what they have to look for, the scientists trained them with representative data they received from their project partners – the Clinic for Bone Marrow Transplants, the Essen University Hospital Institute of Virology, and the Clinic for Internal Medicine I and Institute of Virology at Saarland University. With their simulation tools now trained, the researchers have been working on clinical validation since March 2019. For one year, they will feed in the data of patients currently being treated, compare the results of the simulation, the doctors’ assessments and the actual course of the disease, and thus refine the simulations. In other words, they are retraining the neural networks. “We are certain that these models can offer physicians solid indications of probable complications,” says Kiefer, summarizing their findings to date. Then, so Kiefer hopes, significantly more people might survive this type of transplant.

Early diagnosis by optical coherence tomography

One of the keys to the treatment of cancer – perhaps the most important of all – is the early detection of tumors. Optical coherence tomography (OCT) could play a major role here. Its resolution is between ten and 100 times that of ultrasound. But there is a drawback: the lack of suitable expertise in the day-to-day clinical work in many medical disciplines. Researchers at the Fraunhofer Institute for Production Technology IPT and at Tokyo Women’s Medical University (TWMU) are now hoping to change that with their OCTmapp research alliance. “It is hoped that artificial intelligence will assist doctors with analysis, as well as helping them better understand signal generation – especially the formation of artifacts,” says Niels König, department head at Fraunhofer IPT.

In June, the researchers at Fraunhofer IPT took two OCT systems to Japan, where the preclinical study – testing the technology in relevant applications – will take place. Their aim is to determine what is feasible and how the technology can be used for joint research with Japanese partners. König expects that it will be possible to integrate the system into TWMU’s Hyper Smart Cyber Operating Theater (HyperSCOT) in around five years – and, in the long term, that it will help diagnose cancer e

Certification to ensure a trusted AI

From telecommunications to road traffic, from healthcare to the workplace – digital technology is now an intrinsic part of almost every area of life. Yet how can we ensure that developments in this field, especially those that rely on artificial intelligence (AI), meet all our ethical, legal and technological concerns? In a project led by the Fraunhofer Institute for Intelligent Analysis and Information Systems IAIS, and with the participation of Germany’s Federal Office for Information Security (BSI), an interdisciplinary team of scientists from the Universities of Bonn and Cologne are drawing up an inspection catalog for the certification of AI applications. They have now published a white paper presenting the philosophical, ethical, legal and technological issues involved.


Certified AI applications

Building trust in Artificial Intelligence – interdisciplinary team from IT, philosophy and law defines priorities for the certification of AI.

Fraunhofer Big Data and Artificial Intelligence Alliance

The Fraunhofer Big Data and Artificial Intelligence Alliance consists of 30 institutes bundling their cross-sector competencies. Their expertise ranges from market-oriented big data solutions for individual problems to the professional education of data scientists and big data specialists.

Number of new cancer cases each year in Germany in women and men

Source: German Center for Cancer Registry Data at the Robert Koch Institute, for the year 2013

cancer statistics

Cancer statistics

Estimate of the International Agency for Research on Cancer (IARC) for 2018


people worldwide are stricken with cancer every year.



One in every 5 men and one in every 6 women

will be diagnosed with cancer at some point in their lives.

One in every 8 men and one in every 11 women

die of cancer.



The IARC estimates the yearly increase in the number of new cancer diagnoses worldwide at 




Europe accounts for just 9% of the world population, but 23% of cancer diagnoses worldwide.


people worldwide die of cancer each year.



Contact Press / Media

Janis Eitner

Director Communications

Hansastrasse 27c
80686 Munich, Germany

Phone +49 89 1205-1350

Fax +49 89 1205-771350

Contact Press / Media

Daniela Welling

Head of Corporate Communications

Fraunhofer Institute for Computer Graphics Research IGD
Fraunhoferstr. 5
64283 Darmstadt, Germany

Phone +49 6151 155-146

Contact Press / Media

Katrin Berkler

Head of Communications

Fraunhofer Institute for Intelligent Analysis and Information Systems IAIS
Schloss Birlinghoven
53757 Sankt Augustin, Germany

Phone +49 2241 14-2252

Fax +49 2241 144-2252

Contact Press / Media

Dipl.-Phys. Annette Maurer-von der Gathen

Press and Public Relations

Fraunhofer Institute for Biomedical Engineering IBMT
Joseph-von-Fraunhofer-Weg 1
66280 Sulzbach, Germany

Phone +49 6897 9071-102

Fax +49 6897 9071-188

Contact Press / Media

Susanne Krause M.A.

Public relations

Fraunhofer Institute for Production Technology IPT
Steinbachstr. 17
52074 Aachen, Germany

Phone +49 241 8904-180

Fax +49 241 89046180