However, current augmented reality (AR) technologies, some of which would have seemed worthy of science fiction a decade ago, are being used right now in a variety of healthcare settings to the benefit of both patients and physicians.
Augmented reality is a technology that provides essential new methods of visualization and interaction in the increasingly 3D healthcare sector. With the goals of delivering cutting edge care and staying competitive, healthcare companies across this market segment are investigating how AR can be applied in their respective fields.
In healthcare, AR is making a big difference for medical professionals as they teach and learn, deliver patient care, and ultimately save lives.
Decentralizing care and training
The COVID-19 pandemic demonstrated that many medical procedures and practices can be performed remotely or outside traditional clinical settings. Medical services can take place more “locally” to the patient and AR can help this process change work for both patients and clinicians.
As medical services become more decentralized, more technology is being designed and used to be accessed from any location. Diagnostic tests such as eye exams can use AR to provide real-time evaluations of eye health and vision disorders, not to mention personalized vision correction. Heru, Inc., a healthcare software company and Magic Leap partner, is focused on such innovative diagnostic solutions. The company is working on launching an application on its re:Vive platform on the upcoming Magic Leap 2 headset that is expected to feature several vision diagnostic modalities and reimbursable CPT medical billing codes. This application aims to replace legacy diagnostic devices and include planned applications related to personalized vision-correction that would allow clinicians to diagnose and manage patients with increased efficiency.
Another way that AR can help decentralize the clinical setting is by using it to train the next generation of medical practitioners and provide support to less experienced practitioners. In addition to making instruction accessible from anywhere, having surgical residents practice heart surgery using a computer-generated stent, for example, is safer and much less expensive than having them learn using the real thing.
Radiology has been another pioneering area for AR-based training, which is unsurprising when you consider it’s a field where visualized information is vital. Researchers from Massachusetts General Hospital detailed in a paper for the Radiology journal how learning in AR led to “a higher level of active learner participation, owing to increased social, environmental, and personal presence within the learning activity.”
One of the most immediate and eye-catching uses of AR in healthcare is in surgery. Jennifer Esposito, Vice President and General Manager of Health at Magic Leap, explains how the operating room is the perfect place for augmented reality to make a big practical difference: “The OR is one of the most complex environments in healthcare—or in any industry, really. It’s a highly sensitive environment filled with patient information, equipment, screens, cables, people, and alarms. AR has the ability to tap into this data in ways previously unimagined, to improve the experience for all involved. So, for these reasons, it’s no surprise that the concept of a ‘digital operating room’ has existed for quite a while. The goal has been to alleviate much of this complexity by increasing integration, automation, standardization and replacing some of the physical with digital.”
There are already multiple software solutions that can take data from CT scans, MRIs, and other diagnostic scans and render that data as a 3D model. This can then be viewed by surgeons in an AR setting, showing the precise positions of tumors, organs, and highlighting other key considerations, so that surgeries can be planned in detail before a single incision is actually made.
Such planning has already been utilized in landmark cases like that of Abigail and Micaela Bachinskiy, conjoined twins who were born attached at the head. Surgeons at UC Davis used the Mixed Reality Viewer from Brainlab, running on the Magic Leap AR platform, to study 3D scans of the babies, memorizing the locations of nerves and blood vessels, to prepare for one of the most complex surgeries they had ever performed. The case was so compelling, in fact, that an online video series documenting the process even won an Emmy award.
The use of AR during spinal surgery also has the potential to transform the surgical process and ultimately enhance clinical outcomes. One specific study measured the accuracy of pedicle screw placement in lumbar spine models using a standard surgical navigation platform integrated with the Magic Leap AR platform. The researchers found that combining the advantage of navigation with the 3D awareness of the 360° AR model may provide enhanced intraoperative guidance, which could potentially improve surgical safety, efficiency, ergonomics, costs, and patient safety and outcomes.
Not only can using AR in healthcare reduce the risk to patients and enable surgeons to develop more in-depth plans for the operating room, it also opens up new possibilities for collaborative surgery. Expert specialists can “scrub in” virtually from around the world, observe the operation from the lead surgeon’s point of view, and identify potential issues with their virtual presence as they arise.
This AR-based remote collaboration can also be used to support clinical work such as a specialist consultations from another facility, a medical device expert providing support to facilitate a procedure being conducted using their device, or remote technical support and applications training.
Helping with diagnosis
Augmented reality is also having an impact earlier in patient healthcare journeys. A particularly compelling example of how applied AR can resolve even basic diagnostic problems is Accuvein, a handheld device that scans patients’ arms and then projects a map of their veins onto the skin, allowing nurses to draw blood and insert IV needles with greater precision. Given that almost half of IV attempts miss the vein the first time, this serves as an immediate and obvious benefit to the quality of a patient’s experience.
Another example comes from SyncThink, a neuro-technology company that focuses on eye-tracking analytics. SyncThink is using Magic Leap’s AR-based platform for health assessments, rehabilitation, and sports performance through features like 3D real-world displays, gait, and body positioning. Some of these assessments include balance, proprioception, depth perception, and convergence in patients relating to injuries such as concussions. With this collaboration, they can create specific interactive brain performance training paradigms in the user’s exact environment, rather than trying to produce the same result in a virtual environment.
Existing AR platforms such as Magic Leap can already be used as diagnostic tools. This is because, in order for AR to work, lots of physical information about the user has already been tracked in real-time, such as head movements, eye focus, response times, and gait. An advanced AR platform can detect all of these things and can generate useful medical data.
Jennifer Esposito explains that healthcare startups are already using Magic Leap for just this purpose: “The interesting capabilities of Magic Leap that are inherent in making spatial computing possible are also inherent in enabling different digital diagnostics and therapeutics. So, for example, our eye tracking capability is core to a group of startups that are using it to perform ophthalmology exams, detect concussions, deliver physical therapy, or run brain health or performance measurements.”
Not only can Magic Leap’s AR platform detect and record information relevant to multiple conditions from one wearable headset, its built-in processing and connectivity enables the headsets to be pre-configured to help turn that information into usable visualizations that let multiple practitioners view the results collaboratively.
Visualization is another key area where augmented reality can help with diagnosis. Communication between doctors and patients is a perennial problem, as studies show that between 40% and 80% of medical information that doctors and nurses provide is forgotten by patients immediately. Using an AR platform, patients can be shown visually how their treatment will progress, or where a condition is located in their body, making the information much more memorable—and patients walk away with a clearer understanding of what their doctor is doing and why.
AR is also useful in healthcare for patients with long-term conditions or in longer term therapeutic settings. In the simplest application, AR can enable patients to receive ongoing consultations with their physician without the need for in-person meetings. Doctors can instantly connect with their patients, view collected data from remote monitoring, and even assess the progress of any physical rehabilitation from a first-person perspective.
Magic Leap is already helping to develop platforms to support this kind of work. “Physical therapy rehab applications are really interesting,” says Jennifer Esposito. “The data you're collecting makes the experience more dynamic, so if someone is doing physical therapy, you can see if (and how well) they did the exercise and then have a real-time feedback loop of what they're doing. These are areas where we have partners working with our technology, our platform, to build out these use cases.”
Augmented reality can also make rehabilitative therapy more enjoyable and beneficial for the patient. For example, software company Amblyotech is working with games publisher Ubisoft on an AR package for people suffering from amblyopia, more commonly known as “lazy eye.” Using an AR device to display a different image to each eye adds fun gameplay elements to exercises designed to train the affected eye to align with the other.
There are additional uses outside of physical medicine. The use of AR and VR in treating psychological conditions is well established. In fact, exposure therapy techniques being developed by companies like SmartTek enable those with crippling phobias to encounter the object of their fear in a safe, controlled way, such as letting virtual spiders “crawl” over their arms or being exposed to enclosed spaces. Additionally, therapists can manipulate the distance of walls, speed as they close in, and movement type in treating conditions such as claustrophobia.
AR is also proposed as a treatment for PTSD, so soldiers can relive experiences that left them traumatized until the negative impact is dulled through repetition. This is an area where AR is notably more useful than virtual reality, as full immersion in digital worlds is known to cause motion sickness. By overlaying digital elements with the real world, AR solves that problem at a stroke.
AR in healthcare is already making a difference
Medical researchers have been investigating the practical applications of AR for many years now, and the fruits of their labor are already here. It is no longer unusual for a surgical team to use 3D visualization to better understand a patient's anatomy, or for patient-doctor interactions to be conducted virtually. AR is taking these developments and using visual clarity and connectivity to tie them together and make life easier for staff and outcomes better for patients.The mainstream perception of augmented reality may still be evolving, but in a 2020 Statista study, 38% of industry experts concluded that it is health where the biggest AR disruptions are going to be seen over the coming months, not years. Healthcare professionals are poised to be at the cutting edge of this technological wave, and if the impressive examples we have shared are what’s possible now, imagine where we’ll be in another 10 years.