More than Meets the Eye

In principle, a good surgeon just needs a steady hand and a plan. To cut out a breast cancer tumor, Samilia Obeng-Gyasi uses a pre-op chest mammogram to guide her. It's a standard early-stage surgery called a lumpectomy. Easy, right? She's searching for a stationary target with a known location.

In practice, of course, the process is far from straightforward. A patient's tumor might be located under skin that clothes don't cover, such as near the armpit. To avoid creating exposed scars, surgeons use a technique called tunneling. "You might make an incision around the nipple and tunnel to the area where the tumor is," says Obeng-Gyasi, a surgeon at Indiana University School of Medicine (USA).

Her challenge is to translate a two-dimensional mammogram into a three-dimensional incision-while also considering, in the moment, how to minimize tissue damage and avoid unsightly scarring. To guide the path of the scalpel, she and her colleagues first insert a wire that leads to the tumor. But it's still difficult to make the cut. "You can't see the tip of the wire," says Obeng-Gyasi. To make sure they don't accidentally cut past the tumor, they have to frequently refer back to the patient's mammogram. When instructing medical residents, Obeng-Gyasi will draw visual cues on the patient with a marker.

Surgeons want better visual guidance and feedback, says biomedical engineer Suman Mondal of Washington University in St. Louis (USA). Standard diagnostic imaging, such as x-rays and MRIs, are too bulky to use in the operating room. That's why he and surgeons like Obeng-Gyasi are working together to develop compact techniques to see tumors better during breast cancer surgeries.

To that end, they're experimenting with augmented reality (AR) technology, perhaps most well known for its role in games like Pokemon Go. Unlike virtual reality, AR doesn't block out real surroundings to create an artificial environment. It simply lays extra information on top of the real-world environment. "Augmented reality is basically anything that adds information directly to your visual field," says Mondal.

The added text or graphics can reposition themselves based on the user's point of view. For example, the first AR device, invented in 1968 by Harvard researchers Ivan Sutherland and Bob Sproull, was a bulky cathode-ray tube headset that projected 3D images that could change perspective as users turned their heads. But AR isn't limited to wearable devices. Smartphone and tablet video streams have AR capabilities, too: for example, Pokemon Go overlays cartoon creatures on a smartphone screen depicting surroundings in real time.

The added visuals for medical AR can be very simple, says Lu Lan, a PhD student at Boston University (USA) who collaborates with Obeng-Gyasi. Prototype AR systems overlay 3D lines and shapes onto a video stream or the surgeon's field of view. It could also potentially allow doctors in another room to annotate images for the surgeon in real time.

Roots in Aviation

These surgical-guidance devices actually share similarities with the original intended uses of AR, first developed fifty years ago by researchers employed by the military. The US Air Force's Super Cockpit program, a pioneering effort in the mid-1960s, developed virtual displays to conserve space in cramped cockpits. Their prototype consisted of a headset that could superimpose navigational information in front of the pilot's eyes.

Similarly, medical professionals also need an intuitive way to digest complex information without losing focus on the main task at hand. AR helps surgeons orient themselves, particularly with their depth perception, says professor Jerzy Rozenblit of the University of Arizona (USA). "The analogies to aviation are quite natural," he says.

Optical Fiber Guidewire Locates Tumors

Obeng-Gyasi's group, led by SPIE Member Ji-Xin Cheng of Boston University (USA), designed an AR system with a tablet display adapted from the conventional guide wire that surgeons commonly use. Instead of a regular metal wire, they used a special optical fiber, less than a millimeter in diameter.

Principle of using a fiber optoacoustic guide (FOG) and an AR system to locate the FOG tip and provide the visual guidance on the AR display. Image Credit: Ji-Xin Cheng.

In their design, the surgeon inserts the optical fiber next to the tumor and places three ultrasound sensors on the surface of the breast. They pulse infrared light at hundreds of times per second through the fiber, which causes it to emit ultrasound waves. The ultrasound sensors enable the system to use triangulation to locate the tip of the fiber, whose position is displayed as a green dot on a tablet on a nearby cart. The surgeon consults the tablet while making the cut. "It gives you a way to see the tip of the wire," says Obeng-Gyasi. The fiber and AR system can locate the tumor to 0.8-millimeter accuracy, the team reported in a presentation at Photonics West 2018 as part of the conference on Advanced Biomedical and Clinical Diagnostic and Surgical Guidance Systems. Their work was later published in the journal Light: Science & Applications.

They haven't used the AR system on real patients yet, although Obeng-Gyasi has tested it on a female cadaver. In lieu of a real tumor, her team placed a small metal clip inside the cadaver's breast, and inserted the optical fiber next to the clip. Obeng-Gyasi cut out the clip while using the tablet display as a guide.

Once the device is streamlined for use in the operation room, she says it could cut down the duration of a surgery from half an hour to about 20 minutes.

Potential for Wearables

Lan and Cheng also experimented with a wearable setup: combining the optical fiber with Microsoft HoloLens goggles, where the wearer sees the green dot directly projected onto their visual field. But the HoloLens setup won't be ready for real surgical use in the near term. "It's very hard to convince surgeons that three-pound glasses are the way to go," says Cheng. "They are more comfortable with the tablet."

Still, wearable devices could have their advantages. In particular, surgeons don't need to look away from what they're doing when information is delivered via headset, says Mondal. His group, led by SPIE Fellow Samuel Achilefu at Washington University in St. Louis (USA), has had more success with a wearable design, which they call "Cancer Vision Goggles." They weigh about a third of a pound and are assembled with largely custom-built parts. "They're not as light as reading glasses, obviously, but they're not uncomfortable," says Mondal. "We've had surgeons wear them continuously for 30 minutes without any issues."

A prototype of the "Cancer Vision Goggles" imaging system created by a group at Washington University in St. Louis. Image credit: Suman Mondal and Dr. Samuel Achilefu, Optical Radiology Lab, Department of Radiology, Washington University School of Medicine, St Louis.

Despite the rise of commercial headsets like HoloLens, Mondal thinks that medical AR headsets in the near future will largely be custom built. Off-the-shelf headsets, designed primarily for entertainment and gaming, are too bulky, which also makes them difficult to sterilize for use in the operating room.

Commercial headsets also have potential data security vulnerabilities that might not pass medical privacy regulations. "If you turn on the HoloLens or Google Glass, it connects to the Microsoft or Google database immediately when you turn it on," says Mondal. "There's a chance of patient information and procedures getting online, which is a big insurance risk." For now, they suffice as proof-of-principle demonstrations, but not much else.

Like Cheng's guide fiber, Mondal's headset also helps surgeons better visualize tumors and more clearly see the boundary between tumor and healthy tissue in the breast. About a quarter of breast cancer patients require repeat surgeries because the surgeon fails to remove the tumor in its entirety, says Mondal. So the goal of the headset is to help the surgeon clearly see the tumor's edges and cleanly remove the whole thing.

To use this system, the doctor first injects a fluorescent dye into the patient's tumor. The dye sticks to proteins that proliferate on the surface of cancer cells. Using a tripod-mounted laser, the doctor illuminates the tumor with near-infrared light which causes the tumor to fluoresce. After a sensor on the headset detects the fluorescence, the system positions a false-color image of the tumor onto the patient's body from the surgeon's perspective.

After developing the headset for about a decade, the group has already begun testing their headset on breast cancer patients in clinical trials. Prior to this AR system, they experimented with virtual reality: opaque goggles that live-streamed a video of the glowing tumor's location onto the user's field of view. However, surgeons preferred transparent goggles because they wanted to see the patient in real life, says Mondal.

Clinicians from around the world have contacted the team wanting to try the headset. Currently, the group has two working prototypes, and they're working to make more. Eventually, Mondal wants to fit everything, including the laser, on the headset. "Surgeons are already used to wearing a white light headlamp," he says.

AR Surgical Training Tools

Outside of the operating room, AR technologies could also be used as an educational tool for medical students. Rozenblit's group is developing an AR-based kit that students can use to train their depth perception in surgery. The kits consist of deceptively simple hand-eye coordination tasks students have to accomplish using foot-long metal surgical arms. For example, in one task, a user has to thread a hoop on a thin wire without touching the wire.

Currently, Rozenblit's training kits are limited to laparoscopy, a type of surgery commonly performed in the abdomen or pelvis with the aid of a camera inside the body. Similar kits could be developed for breast cancer surgery training, says Rozenblit.

AR-based training can also introduce students to high-risk tasks that their predecessors may have encountered only in real surgeries without prior preparation. "We can create all sorts of training scenarios that are highly repeatable," says Rozenblit. Students can gain experience dealing with rare but critical situations without the risk.

Rozenblit's computer-assisted surgical trainer can provide visual, force, or audio guidance for realistic training tasks.

But AR still has its technical limitations. Users have reported eye fatigue after using headset-based systems for too long. Another general challenge for the field is to improve the position accuracy of the image projection, generally known as image registration. The headsets often place images in the wrong place, says Mondal. To improve this, researchers are trying to better track the user's eye movements.

In addition, some interactive controls are difficult to use. A surgeon wearing a headset may want to toggle between different displays. One method is to gesture in front of a sensor on the headset, but the movements are not intuitive, says Lan.

Most AR developments for breast cancer surgery are still in the prototype stage. But Lan, Cheng, and Mondal are also working to commercialize their systems. To that end, Cheng has co-founded an Indiana-based company, Vibronix, and they are working to secure FDA approval for their optical fiber system. Achilefu's team, including Mondal, is currently courting investors to build their headset at scale. They also plan to sell a tumor dye to accompany the goggles.

Solving One Surgical Problem May Solve Another

Although the techniques seem to be tailored to specific surgical tasks, many are easily adaptable for use in the treatment of other diseases. Cheng, for example, wants to adapt his breast cancer guide fiber for robot-aided laparoscopic surgery, where the AR visualizations are translated into robot instructions.

Conversely, AR techniques for treating other diseases might be useful for breast cancer surgery, too, says Mondal. For example, breast cancer surgeons might eventually be able to use precision techniques originally developed for improving brain surgery outcomes. Brain tumor removal requires unique care because even small mistakes can damage the patient's basic cognitive functions. "When they go to take out the tumor, they have to make this sort of cost-benefit decision," says Mondal. "They might ask, do I take out this tumor or do I leave it in because it's impinging on the visual cortex center?" Researchers are testing AR techniques that give the surgeon feedback to make these decisions.

AR's overarching benefit is to provide surgeons with more objective guidance. Currently, to cleanly cut out a tumor, breast cancer surgeons might try to figure out its boundaries by pressing on the tissue. Tumor tissue is generally harder and more fibrous than healthy tissue. "That's pretty subjective," says Mondal. "There's a lot of variability, surgeon to surgeon."

AR displays provide image-based feedback in real time-reducing the need for surgeons to rely on their intuition. "Surgeons want something to help them see," says Cheng. With augmented reality technology, they may have found a solution.

Sophia Chen contributes to Wired, Science, and Physics Girl. She is a freelance science writer based in Tucson, Arizona.