Remote operations get closer with AI that looks at shapes instead of pixels

Concerns about the shortage of medical personnel are increasing worldwide. Artificial intelligence (AI), developed by a research group at TU Darmstadt, can come to the rescue of less qualified personnel by helping them perform screenings and even operations in collaboration with a remote expert. “With a stable internet connection, our AI can help patients in rural areas around the world,” says Anirban Mukhopadhyay, computer scientist at TU Darmstadt.

Research recently showed that the shortage of healthcare personnel in the Netherlands alone is expected to increase to 135,000 by 2031. The shortages occur throughout healthcare, including the operating room.

But the staff shortage extends beyond the Netherlands. It is a global problem that mainly affects developing countries, says Mukhopadhyay. He is originally an Indian computer scientist and currently heads a research group at the TU Darmstadt that focuses on AI in medical applications. “I have lived in India for over 20 years and experienced health care up close. I’ve seen people close to me die because staff couldn’t figure out exactly when a patient in the ICU was infected with COVID-19. Once you’ve seen that, you know there are serious problems. Outside the major urban centers, the country has a serious shortage of qualified personnel.”

Anirban Mukhopadhyay

Shapes replace pixels

Technologies such as surgical robots and artificial intelligence play a key role in solving the staffing problem, is Mukhopadhyay’s belief. These kinds of solutions are already being used here and there. The Maastricht hospital, for example, works with the Da Vinci Robot, a surgical system that uses a minimally invasive surgical approach. A surgeon works with a console that is almost completely separate from the robot.

With the help of an AI that ‘understands’ the procedures and a stable network connection, operations could eventually even be performed while the patient is at a distance. Mukhopadhyay’s research group is well on its way to making that dream come true with the help of an artificial intelligence that works differently from the mainstream AI currently used in medical care.

“Our AI works on the basis of shapes rather than pixels,” the researcher explains. “In the medical world, we normally have to make do with a very limited amount of data and no two operations are identical. An AI looking at pixels needs to cover a lot of search space. But if we include the shape knowledge of organs and tissues in our algorithms, you narrow that search space. That way you don’t need a lot of data to get a good picture of bones and structures in a body, and to know how a patient is doing.”


The AI ​​developed in Darmstadt can be used for screening purposes, among other things, to support less qualified personnel in remote rural areas in developing countries. “There may be a huge shortage of doctors in rural India, China and Africa, but mobile networks are available in increasingly remote areas. We want to make good use of that,” says Mukhopadhyay.

Take a tuberculosis patient from India. Getting a patient’s sputum to the nearest testing center, sometimes more than 500 kilometers away, is a huge logistical challenge. “In this case, it works much better to drive a van past villages, capture X-ray images and send them via a mobile network. An AI can then perform a remote screening based on these images.”

heart attack

Artificial intelligence improves treatment of myocardial infarction in women

Heart attacks are one of the most common causes of death worldwide. Women who have a heart attack have a higher death rate than men.

Sub-millimetre work

The AI ​​also comes in handy in very precise operations that require very precise work, such as in neurosurgery or when placing hearing prostheses in the inner ear. “Before a prosthesis is placed, the skull must first be opened to see what a patient’s anatomy looks like. This is necessary, because the ‘real’ operation has to be done very carefully; There are several nerve pathways leading to the ear that should not be touched. Otherwise, the face can partially paralyze the patient.”

“Surgical robots are accurate and minimally invasive. But these robots have to be navigated without a direct line of sight. We’re dealing with a chicken-and-egg problem here, because the navigation AI needs training data, and the robot needs trained AI to navigate. Our solution is to use preoperative computed tomography scans of the head to train the AI. This makes it possible to provide insight into nerve pathways, based on little data. In this way we ensure a safe way of working with as little strain as possible for the patient.”

A long way to go

The research group still has a long way to go before the innovation can actually be applied in the medical world. “Patient safety first” is our motto in translational research. We need to be sure that all systems work together securely. Translating research from surgical robotics to the clinic takes time.” However, the researcher does see rapid progress for applications in the field of screening and diagnosis.

More and more applications

In the coming years, the medical world will use AI solutions for a broad spectrum of purposes, predicts Mukhopadhyay. “Whether it’s screening patients in Africa, assisting surgeons in precision surgery or, in the (distant) future, even coming to the aid of doctors when they need to write a patient report. Healthcare will undergo radical changes in the coming years as a result of new technologies such as AI. It is a long battle to give everyone access to care. But it’s one worth fighting for.”

The article is in Dutch

Tags: Remote operations closer shapes pixels

PREV This is Charlotte Labee’s food diary
NEXT This is the reason you sometimes sleep badly after drinking alcohol