[2210.14349] A DirectX-Based DICOM Viewer for Multi-User Surgical Planning in Augmented Reality

Preoperative medical imaging is an essential part of surgical planning. The
data from medical imaging devices, such as CT and MRI scanners, consist of
stacks of 2D images in DICOM format. Conversely, advances in 3D data
visualization provide further information by assembling cross-sections into 3D
volumetric datasets. As Microsoft unveiled the HoloLens 2 (HL2), which is
considered one of the best Mixed Reality (XR) headsets in the market, it
promised to enhance visualization in 3D by providing an immersive experience to
users. This paper introduces a prototype holographic XR DICOM Viewer for the 3D
visualization of DICOM image sets on HL2 for surgical planning. We first
developed a standalone graphical C++ engine using the native DirectX11 API and
HLSL shaders. Based on that, the prototype further applies the OpenXR API for
potential deployment on a wide range of devices from vendors across the XR
spectrum. With native access to the device, our prototype unravels the
limitation of hardware capabilities on HL2 for 3D volume rendering and
interaction. Moreover, smartphones can act as input devices to provide another
user interaction method by connecting to our server. In this paper, we present
a holographic DICOM viewer for the HoloLens 2 and contribute (i) a prototype
that renders the DICOM image stacks in real-time on HL2, (ii) three types of
user interactions in XR, and (iii) a preliminary qualitative evaluation of our
prototype.