A new reality in surgery through augmented reality

Marc Katz, M.D., touches the man’s heart, ever so gently. Then he takes his index finger and thumb, presses them against the spot he’s interested in and … he pinches.

To zoom out. 

Thankfully, the cardiothoracic surgeon isn’t handling a real heart, rather, it’s an augmented reality (AR) model of a real heart, using a holographic headset manufactured by Microsoft and imaging software created by Virginia-based ClearView Surgical. If the surgeon and tech companies have their say, this is the future of surgery – and it’s not that far away. 

Most people have heard of virtual reality (VR), a completely immersive experience that replaces your real-life environment with a simulated one. It’s everywhere these days: movies, video games, real estate. AR is slightly different. Instead of using a simulated environment, AR adds digital elements to your real-world surroundings. It, too, is starting to creep into everyday life. Want to see how that couch looks in your living room? Open up an app and you can see for yourself. That’s AR. 

As Katz manipulates the organ in all different directions, he looks like someone trying to walk in a dark room – hands out in front, gently feeling for furniture or walls. But from Katz’s perspective, thanks to AR, what he sees is a very real looking heart hovering above his second-floor office desk. 

How it works is – and this is radically oversimplified – the engineers at ClearView Surgical build exact 3D models of patients’ hearts (though it could be any organ; Katz said the heart is one of the more complicated ones because it is always moving), using the information from CT scans. In turn, these models then allow the surgeon to manipulate the heart in any direction. Need to remove a valve? No problem. What’s behind that ventricle? Let’s take a look. By marrying technology with medicine, surgeons like Katz will one day be able to look at – and touch – a patient’s heart without actually opening them up. The result: zero risk, tons of reward. 

Though not yet used in actual practice (Katz and ClearView Surgical seem to think we’re still at least a year away), an idea that once seemed like a Hollywood fantasy is now a heck of a lot closer to becoming science nonfiction.

“Imagine all the things this technology will allow us to do,” Katz said. Take the heart apart, turn it upside down, you name it –it’s possible, he said. All without endangering the patient. Katz even thinks the technology could eventually lead to performing mock surgeries, which would allow the surgeon the luxury of experimentation.

Dan Neuwirth, co-owner of ClearView Surgical, said that despite radical improvements in imaging technology over the years, how doctors view the images has remained largely unchanged since the 1950s. For the past 70 or so years, folks in the medical field have looked at those images on film or computer screens. In other words, it’s all 2D. Sure, technology has allowed for simulating three dimensions, but in the end, the image the doctor is looking at is still flat. 

Neuwirth and fellow co-owner Daniel Salzberg knew there was a tremendous opportunity, utilizing current technology, to change that. So they approached Katz and a handful of other surgeons across the country and asked for feedback. It’s a partnership where each side provides necessary value. Katz provides the imaging and the surgical know-how, ClearView Surgical makes the augmented version – so surgeons can manipulate it to their, well, heart’s content.

“With Dr. Katz, we get to tap into the mind of a super accomplished surgeon and together, we can come up with ways to improve patient outcomes,” said Salzberg. 

“It really has been a great partnership. Dr. Katz is a very forward-thinking guy in terms of technology,” Neuwirth added. 

Neuwirth and Salzberg see this as the tip of the iceberg when it comes to utilizing AR in medicine. 

“Obviously this is a great help for really complex cases like Dr. Katz works on,” Neuwirth said. “That’s one end of the spectrum. But we also see this eventually helping out residents with commonplace surgeries as well.”

From a medical standpoint, Katz sees this as a first step in what he hopes will be a wide range of medical applications. 

“Imagine being in the OR and taking the images we have and fusing them with three-dimensional images,” Katz said. “It would be like looking through someone’s body like I’m wearing X-ray glasses.”