AR and 3D simulations project desired result of surgery on patients face

Koichi Ueda, MD, PhD, Daisuke Mitsuno, MD, and colleagues of Osaka Medical College, Japan, have developed and experimented with an augmented reality (AR) system for evaluation of improvements of the body surface, a key consideration in plastic surgery. Initial experience in eight cases suggests that augmented reality could be a useful guide to planning, performing and evaluating the results of facial reconstruction and other procedures. A paper regarding the project has been published in Plastic and Reconstructive Surgery—Global Open, the official open-access medical journal of the American Society of Plastic Surgeons (ASPS).

Simple and modifiable AR technique

Drs. Ueda and Mitsuno explain they sought to develop a sophisticated yet simple and modifiable AR technique for use during plastic and reconstructive surgery. For this, the researchers used a high-definition digital camera to capture 3D image of the facial surface and computed tomography scans to obtain digital information on the underlying facial bones for each patient. These digital data were then manipulated to create 3D simulations of the ideal final results. For example, in a patient with a fractured cheekbone, the reconstruction was simulated by obtaining and reversing an image of the opposite, uninjured bone.

Using a pair of commercially available smart glasses, the surgeon was able to superimpose the 3D digital simulation image of the desired appearance over the patient’s face during surgery. The group also used free, open source software products to solve various technical problems, including manipulating and displaying the 3D simulations and lining them up (registration) with the surgical field.

AR systems helps in planning reconstruction

Preliminary experience with the AR system was gathered with in eight patients undergoing reconstructive facial surgery. The AR system helped in planning and confirming reconstruction of the underlying facial bones, for example, in a patient with a congenital bone development disorder and another patient with a complex facial fracture. In all cases, the 3D simulation of the body surface provided a visual reference of the final facial appearance.

With each new case, the group made additional technical refinements to address limitations of the AR system. Although the experimental system was not actually used to guide surgery in this initial experience, it helped in visualizing the planned correction and confirming the final outcome. “In all cases in this study, the body surface contour after the procedure and the ideal postoperative image almost coincided,” Dr. Ueda adds.

Further studies to confirm benefits

The research group now plans further studies to confirm the benefits of using the AR system during plastic and reconstructive surgery and to refine the display method for comparing before and after images. Future innovations may include the ability to quantitatively evaluate the improvements in body surface and even to enable simple navigation of internal organs.

Drs. Ueda and Mitsuno believe AR technology could become a useful tool for teaching surgical skills.
“Our findings are not only useful for body surface evaluation but also for effective evaluation of AR technology in the field of plastic surgery.”