Back to Top

Pakko De La Torre // Creative Director

VTK.js Transforms Web-based Visualization with Immersive Virtual and Augmented Reality

VTK.js Transforms Web-based Visualization with Immersive Virtual and Augmented Reality

Are you looking to transform your web pages into immersive experiences? Kitware can help bring your medical and scientific data off the computer screen and into real-life using VTK.js, our open source rendering library for scientific visualization on the web. VTK.js now has modern web-based virtual and augmented reality visualization, thanks to the power of WebXR.

A closer look at medical surface data in augmented reality with VTK.js

Web-Based Immersive Rendering

WebXR is a browser technology that exposes an API to enable tracking and rendering mixed reality sessions on mobile or head-mounted devices, such as your smartphone or a dedicated virtual reality headset. By integrating WebXR functionality into VTK.js, we leverage the existing rendering pipeline with WebGL (and soon, WebGPU) to bring new, immersive visualizations to customers across a broad array of hardware.

An expanding set of AR and VR examples are available as part of the VTK.js documentation. Additionally, all of the previous VTK.js VR examples have been updated for WebXR. Dedicated hardware such as a head-mounted display or a mobile device can be used to try out our VR and AR examples. Developers can also install the WebXR Emulator extension for Chrome or Firefox to simulate XR hardware in their browser.

VTK.js has a growing set of WebXR examples and documentation

Virtual Reality Capabilities

WebXR succeeds the deprecated WebVR and Cardboard VR APIs that were previously supported by VTK.js. All of the previous VTK.js VR examples have been updated for WebXR. Additionally, the WebXR polyfill can be dynamically included to enable backward-compatible support for these deprecated APIs. Moving forward, any native VR backend that implements the OpenXR standard, including OVR and SteamVR, can be used to drive WebXR content from VTK.js.

Virtual reality rendering is accomplished in several stages:

These details are all hidden in the standard VTK.js WebGL RenderWindow, which makes it easy to add VR support to existing and new projects by simply invoking the “startXR” method. Additionally, VR controller events such as trigger presses are captured with the standard VTK.js RenderWindowInteractor and can be handled with an InteractorStyle or directly in an application.

VTK.js VR Cone Example with the WebXR Emulator Browser Extension

Augmented Reality Capabilities

WebXR introduces web-delivered augmented reality support into VTK.js for the first time, initially targeting mobile handheld devices. The ARCore backend drives AR experiences on Android devices and is available in the Chrome mobile browser. At the time of writing, development is still underway on WebXR’s ARKit browser compatibility on iOS devices, but iOS users can download a dedicated application such as the Mozilla WebXR Viewer to engage with WebXR content.

Much like VR rendering, AR rendering relies on pose information delivered through the WebXR API to set the virtual camera position in the virtual world for rendering with the VTK.js pipeline. WebXR hides the camera stream from the consuming application and presents it instead as a skybox with virtual objects rendered in front, with the requirement that any virtual background is first made transparent.

Projects already relying on the VTK.js WebGL RenderWindow can once again easily take advantage of the new functionality by invoking “startXR(true).” AR events can be queried directly through the WebXR Gamepad API or handled as events from RenderWindowInteractor.

To showcase the VTK.js’ AR capabilities, we have recently added VTK.js example applications for viewing volumes and geometries in AR.

Viewing medical data in augmented reality with the VTK.js GeometryViewer example

Potential Applications

Initial WebXR integration with VTK.js has been funded by the NIH through multiple awards. This work is freely available for academic and commercial efforts. Web-delivered immersive visualization with VTK.js has many potential applications in a variety of fields, such as those described below:

3D geometries can be shown at scale, to confirm form, function, and fit

Partnering with Kitware

Kitware is dedicated to advancing cutting-edge scientific computing and visualization technologies. Our team is composed of computer scientists, software engineers, and scientific computing experts. They have the knowledge and expertise to support your pursuit of AR and VR technology and can help you practically incorporate these XR experiences into your work. Whether you’re a research center, university, or commercial company, partnering with Kitware will ensure you effectively leverage this technology. For more information, you can send us a message at or visit our commercial projects page. 

Research reported in this publication was supported by the National Institute Of Biomedical Imaging And Bioengineering and the National Institute of Neurological Disorders and Stroke of the National Institutes of Health under Award Numbers R01EB028283, R01EB014955, and R42NS086295. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

This content was originally published here.