Just a decade ago, the idea of incorporating spatial computing systems into medicine seemed like a futuristic dream. Today, more than 120 Magic Leap 1 headsets with the Mixed Reality Viewer from Brainlab are in clinical use around the globe. These highly advanced visualization tools have a variety of uses, helping medical professionals with anything from surgical planning and medical training to providing patients with an easier way to see and understand their diagnosis and treatment.
Mixed Reality Viewer on Magic Leap 1 turns any work area into a virtual, interactive space strengthening 3D visualization, interaction and understanding. Built in sensors in the Magic Leap 1 map the room, so clinicians can realistically mimic the planned surgery, placing 3D anatomical models and 2D image slices anywhere in the space, whether that’s an office, a meeting room or even at home. The system allows for up to four people to collaborate and interact with each other in real time, either in the same room or even remotely within the same network.
“We’ve found that we can effectively lower the risk of surgical complications for almost any case when using the Mixed Reality Viewer software on Magic Leap 1 during planning,” explained Veit Braun, MD, director and professor at the Jung-Stilling Hospital, part of the Diakonie Hospital network in Siegen, Germany. “We see fewer of the distinct complications we saw before using this technology. It also helps shorten surgical times and patient time under anesthesia which is a great benefit.”
Surgical planning is one area in which spatial computing is becoming a staple for viewing and interacting with patient imaging and data. Clinicians are identifying significant value in surgical precision and time savings from using Mixed Reality Viewer on Magic Leap 1 to support treatment planning. The technology is already in daily clinical use in Europe for multiple treatment indications including aneurysms and skull base tumors. Surgeons are increasingly incorporating Mixed Reality Viewer from Brainlab on Magic Leap 1 into their clinical routine for cervical fractures and craniomaxillofacial surgery given the complexity and need for detailed planning. The ability to enlarge the anatomy and to virtually walk around, or even inside of it, delivers a wholly new perspective that is so close to reality and so vivid that the industry will rapidly realize a paradigm shift over the next few years.
How AR Surgical Planning Actually Works
To enable viewing a case in spatial 3D, the surgeon begins by creating a plan using surgical planning software. A 3D model of the patient’s anatomy is generated from available radiology imaging, like MRI and CT. The software then identifies and outlines the various anatomical structures and the surgical target, like a tumor or an arteriovenous malformation (AVM), on the image.
With these structures identified in the 3D model, the surgeon can begin to plan the optimal approach to the surgical target, avoiding any critical anatomy. Once this plan is created, the surgeon can then put on an augmented reality head-mounted display, such as the Magic Leap 1, and move it from the workstation’s 2D screen and place it into the room. They can walk around it, assessing the plan from all angles, and even invite other clinicians to join in real-time to review and discuss the plan together.
Even though this technology is considered the pinnacle of surgical plan review today, the concept of assessing the surgical situation virtually actually dates back more than fifty years. The first hints of using a spatial viewing technology in surgery began in orthopedics when Robert Mann proposed using VR to allow surgeons to test multiple approaches for orthopedic cases in 1965.i Fast forward to 2020, this technology is more real than ever as it expands into a wide range of surgical subspecialties.
Spatial Computing in Neurosurgery
The close proximity of critical structures in the brain makes neurosurgery exceedingly complex. One small deviation from a surgical plan can drastically affect the surgical outcome and even the patient’s quality of life. Technology, like augmented reality, that allows the surgeon to review a detailed plan in spatial 3D before surgery is a crucial advancement in ensuring the treatment is carried out exactly as intended.
“I do every aneurysm case using Brainlab Mixed Reality Viewer on Magic Leap 1 during the planning process. It helps me easily determine if I need to do clipping or coiling. Viewing in mixed reality is especially helpful for clipping. The technology shows you exactly where the aneurysm and aneurysm neck are located. Not only is it extremely helpful when planning the approach, it’s also fascinating to virtually view and interact with the anatomy.” —Prof. Veit Braun, MD, Head of Neurosurgery at Diakonie Klinikum Jung-Stilling, Siegen, Germany
Another clear advantage of planning with an Augmented Reality tool, like Brainlab’s software on Magic Leap 1, is that from the first cut, the surgeon knows exactly what they are going to do. The surgery can be completed more quickly and with a decreased risk of complications. Since prolonged time under anesthesia increases the risk of complications during and after the procedureii, more efficient surgeries benefit the patient immensely.
Interdisciplinary planning for CMF cases with Augmented Reality
For craniomaxillofacial (CMF) surgery, incorporating the use of Augmented Reality technologies into the planning process supports clinicians in their decision-making. With Magic Leap 1 multiple clinicians can join a session and observe the same patient images at the same time—even remotely. This means that the technology actually facilitates interdisciplinary planning.
“I believe that augmented reality applications will be the next big step in virtual surgical planning, patient education, as well as medical education. Augmented reality, combined with planning tools, allows craniomaxillofacial surgeons to segment out and visualize complex anatomy in a three-dimensional environment, prior to stepping into the operating room. The fact that multiple users can interact with these virtual objects in the same environment is a game changer. We are very excited about the opportunities this will provide at our institution.” – Prof. Bradley Strong, MD, Otolaryngologist at University of California – Davis Medical Center.
Surgeons, radiologists, and radiation oncologists, for example, can discuss resection with adjuvant radiotherapy during a tumor board meeting to determine the best approachiii. Since the surgeon knows and has seen the extent of resection in advance, the surgery can progress quickly as they know when to stop and in turn, move the patient to radiation therapy.
Functional Stereotactic Surgery
Functional neurosurgery applications, such as deep brain stimulation (DBS) and stereo-electroencephalography (SEEG), require interdisciplinary coordination in many steps of the clinical workflow. Neurosurgeons, epileptologists and neurologists need to work together from preoperative planning all the way to postoperative patient management.
Augmented Reality technology supports functional neurosurgeons and epileptologists in combining their knowledge to create an optimal trajectory plan before an SEEG case. These types of cases consist of the implantation of up to 20 electrodes to identify epileptic foci in drug-refractory epilepsy patients. Collaboration between these experts can contribute to improved outcomes.
“Many SEEG cases are three-dimensionally complex implantations that are planned in an interdisciplinary team. Since the implantations are tailored according to clinical, imaging and EEG findings, each scheme is different. Especially for this use-case the visualization in mixed reality turns out to be very helpful for the team discussion during planning.” – Peter Reinacher, MD, Consultant, Department of Stereotactic and Functional Neurosurgery, Medical Center – University of Freiburg, Germany, and Group Leader (ATTRACT), Fraunhofer Institute for Laser Technology, Aachen, Germany.
Hours or days after deep brain stimulation surgery, the electrodes are programmed by neurologists to ensure they stimulate the patient’s brain as planned. This technology can uniquely contribute to their understanding of the patient’s anatomy on a new level, supporting visualization-based programming in place of traditional trial and error approaches.
The 3D reconstructions the clinician can view in a spatial computing headset such as Magic Leap 1 greatly assists in their understanding of very complex anatomy. An example from vascular surgery would be a deep-seated kidney artery aneurysm, in which a true-to-life representation provides a much clearer picture into the case than two dimensional scans aloneiv.
In the future when spatial computing and related technologies come into regular use intraoperatively, the surgeon will have access to a range of data, without ever having to look away from the surgical situation. Bloodwork results, stent graft measurements, EKG, preoperative images, or even the radiation exposure to the surgical team could be displayed, turning the surgeon’s field of view into a “virtual cockpit”v.
Spatial Surgery is Both the Future and the Present
In neurosurgery, CMF, functional and vascular surgery today, augmented reality and spatial computing technologies are already in use and increasingly becoming an essential tool to better understand anatomy and ensure the best possible surgical plans.
Since spatial computing technologies are adaptable to changes and new developments, they will continue to play a significant role in surgery in the future. Incorporating these advanced visualization tools into surgical workflows is already an easy step clinicians can take today to ensure their patients have great surgical outcomes.
Spatial computing also has many use cases outside of surgery that have the potential to transform how healthcare is delivered and how patients engage with their own care. There are more than 10 clinical studies underway by Magic Leap partners working on Magic Leap 1 in use cases outside the operating room.
i Rosen JM. Virtual Reality and Medicine-Challenges for the Twenty‐First Century. Wiley Online Library. https://onlinelibrary.wiley.com/doi/abs/10.1002/0471216690.ch4. Published October 29, 2001. Accessed October 20, 2020.
ii Phan K, Kim JS, Kim JH, et al. Anesthesia Duration as an Independent Risk Factor for Early Postoperative Complications in Adults Undergoing Elective ACDF. Global spine journal. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5721997/. Published December 2017. Accessed October 20, 2020.
iii CMF PAPER BEING PUBLISHED
iv Böckler, D., Geisbüsch, P., Hatzl, J. et al. Erste Anwendungsoptionen von künstlicher Intelligenz und digitalen Systemen im gefäßchirurgischen Hybridoperationssaal der nahen Zukunft. Gefässchirurgie (2020). https://doi.org/10.1007/s00772-020-00666-9
v Böckler, D., Geisbüsch, P., Hatzl, J. et al. Erste Anwendungsoptionen von künstlicher Intelligenz und digitalen Systemen im gefäßchirurgischen Hybridoperationssaal der nahen Zukunft. Gefässchirurgie (2020). https://doi.org/10.1007/s00772-020-00666-9