Malignant tumors often form metastases that spread to other parts of the body via the lymphatic system. High surgical skills are required to identify the precise location of the affected lymph nodes, enabling them to be completely removed. 3D-ARILE supplements these skills because the AR glasses enable the doctor to precisely locate the site of the lymph node during surgery. The new system will be presented to the public at the Medica trade fair, from November 13 to 16.
Cancer cells can be transported by the lymph fluid to the lymph nodes. This leads to the growth of secondary tumors, or metastases. The first lymph nodes to be affected are sentinel lymph nodes. They are the first to receive lymphatic drainage from a malignant, primary tumor, and if they contain cancer cells it’s likely that other metastatic tumors have begun to grow elsewhere.
Sentinel lymph nodes therefore play a primordial role in the diagnosis and therapy of certain types of cancer such as skin, breast or prostate cancer. After excision of the tumor, a lymph node biopsy is carried out to determine whether metastasis has taken place. But despite all medical advances it is still difficult for doctors to determine the precise anatomical location of sentinel lymph nodes during surgery and to check that the affected lymph nodes have been completely removed.
The novel augmented reality system, based on the use of data glasses, helps surgeons locate lymph nodes with the aid of virtual markers. What makes these AR glasses special is that they work in combination with powerful medical navigation software, a stereoscopic, near infrared (NIR) camera system, and indocyanine green (ICG) fluorescent dye.
“To make the affected lymph nodes visible, the fluorescent dye is injected into the patient in the direct vicinity of the primary tumor. It then spreads along the lymphatic pathways and collects in the sentinel lymph nodes,” explains Fraunhofer IGD research scientist Dr. Stefan Wesarg.
The dye fluoresces when exposed to infrared light, in this case generated by infrared LEDs. NIR cameras capture the fluorescence and produce a 3D reconstruction of the affected lymph node. This virtual image showing their exact position is superimposed in real time and can be viewed by the surgeon through the data glasses.
The necessary software was developed by the research team in Darmstadt. “In our case, the malignant tissue is marked in green. In this way, the surgeon can verify that every last trace has been removed,” says Wesarg.
The data glasses, by contrast, display the site of the affected lymph node instantly. This greatly relieves the stress on the OR staff, who no longer need to watch an additional monitor and compare the image on the screen with that on the camera’s display. “As a result, the doctor can focus entirely on the patient and carry out surgeries without hassle,” explains Wesarg.
Another advantage of the AR glasses is that they are light and extremely comfortable to wear, as dermatologists at Essen University Hospital have confirmed after carrying out numerous tests. The partners who developed the system remained in close contact throughout the project, enabling them to design a navigation aid optimized to the surgeon’s needs.
The Fraunhofer IGD research team was responsible for developing the software. It includes an image processing system which detects fluorescent light emitted by the lymph nodes, uses this data to calculate their 3D coordinates, and displays their position in the data glasses. The software also performs the necessary calibration of the hardware system. The entire hardware can be controlled via 3D-ARILE. The system also includes the user interface for the operating surgeon.
Large increase in skin cancer
According to the German Federal Statistical Office, the number of patients treated for skin cancer in German hospitals has grown significantly in recent years. Malignant melanoma, or black skin cancer, is the moest feared diagnosis.Cancer cells can be transported by the lymph fluid to the lymph nodes. This leads to the growth of secondary tumors, or metastases. The first lymph nodes to be affected are sentinel lymph nodes. They are the first to receive lymphatic drainage from a malignant, primary tumor, and if they contain cancer cells it’s likely that other metastatic tumors have begun to grow elsewhere.
Sentinel lymph nodes therefore play a primordial role in the diagnosis and therapy of certain types of cancer such as skin, breast or prostate cancer. After excision of the tumor, a lymph node biopsy is carried out to determine whether metastasis has taken place. But despite all medical advances it is still difficult for doctors to determine the precise anatomical location of sentinel lymph nodes during surgery and to check that the affected lymph nodes have been completely removed.
Navigation aid for lymph node removal
3D-ARILE, developed by researchers at the Fraunhofer Institute for Computer Graphics Research IGD, provides doctors with a navigation aid for lymph node removal. The Clinic for Dermatology at Essen University Hospital and Trivisio Prototyping GmbH collaborated with the Fraunhofer researchers on this project.The novel augmented reality system, based on the use of data glasses, helps surgeons locate lymph nodes with the aid of virtual markers. What makes these AR glasses special is that they work in combination with powerful medical navigation software, a stereoscopic, near infrared (NIR) camera system, and indocyanine green (ICG) fluorescent dye.
“To make the affected lymph nodes visible, the fluorescent dye is injected into the patient in the direct vicinity of the primary tumor. It then spreads along the lymphatic pathways and collects in the sentinel lymph nodes,” explains Fraunhofer IGD research scientist Dr. Stefan Wesarg.
The dye fluoresces when exposed to infrared light, in this case generated by infrared LEDs. NIR cameras capture the fluorescence and produce a 3D reconstruction of the affected lymph node. This virtual image showing their exact position is superimposed in real time and can be viewed by the surgeon through the data glasses.
The necessary software was developed by the research team in Darmstadt. “In our case, the malignant tissue is marked in green. In this way, the surgeon can verify that every last trace has been removed,” says Wesarg.
Fluorescent dye alternative to radioactive nanocolloids
Until now, doctors used technetium-99m as a medical tracer. By replacing this radioisotope with ICG fluorescent dye, patients can be treated without risk of radiation side-effects. Time, according tot he research team, is another essential factor. The level of radiation emitted by lymph nodes containing the radioactive marker is very low. This made it necessary to use scintillation (or gamma) cameras that required around 30 minutes to capture the images needed to determine the precise location of a lymph node.The data glasses, by contrast, display the site of the affected lymph node instantly. This greatly relieves the stress on the OR staff, who no longer need to watch an additional monitor and compare the image on the screen with that on the camera’s display. “As a result, the doctor can focus entirely on the patient and carry out surgeries without hassle,” explains Wesarg.
Another advantage of the AR glasses is that they are light and extremely comfortable to wear, as dermatologists at Essen University Hospital have confirmed after carrying out numerous tests. The partners who developed the system remained in close contact throughout the project, enabling them to design a navigation aid optimized to the surgeon’s needs.
3D-ARILE
The augmented reality system comprises hardware and software components. The hardware was developed by Trivisio Prototyping GmbH. In addition to the data glasses with an integrated camera and two displays, designed specifically for medical applications, the company also developed two infrared and two visual cameras. These four optical devices are integrated in a cube-shaped unit suspended over the operating table.The Fraunhofer IGD research team was responsible for developing the software. It includes an image processing system which detects fluorescent light emitted by the lymph nodes, uses this data to calculate their 3D coordinates, and displays their position in the data glasses. The software also performs the necessary calibration of the hardware system. The entire hardware can be controlled via 3D-ARILE. The system also includes the user interface for the operating surgeon.
