Skip to content

Latest commit

 

History

History
256 lines (182 loc) · 10.3 KB

README.md

File metadata and controls

256 lines (182 loc) · 10.3 KB
Raw
layout permalink redirect_from project_title key_investigators
pw39-project
/:path/
/PW39_2023_Montreal/BreakoutSessions/RenderingBreakout/Readme.html
/PW39_2023_Montreal/BreakoutSessions/RenderingBreakout/README.html
The future of rendering in VTK and Slicer
name affiliation Country
Simon Drouin
École de Technologie Supérieure
Canada
name affiliation country
Steve Pieper
Isomics, Inc.
USA
name affiliation country
Murat Maga
University of Washington
USA
name affiliation country
Andras Lasso
Queen's University
Canada
name affiliation country
Sara Rolfe
Seattle Children's Research Institute
USA
name affiliation country
Jean-Christophe Fillion-Robin
Kitware, Inc.
USA
name affiliation country
Stephen Aylward
Kitware, Inc.
USA
name affiliation country
Rafael Palomar
NTNU
Norway
name affiliation country
Sankhesh Jhaveri
Kitware, Inc.
USA
name affiliation country
Matt McCormick
Kitware, Inc.
USA
name affiliation country
Forrest Li
Kitware, Inc.
USA
name affiliation country
Lucas Gandel
Kitware SAS
USA
name affiliation country
Jaswant Panchumarti
Kitware, Inc.
USA
name affiliation country
Shreeraj Jadhav
Kitware, Inc.
USA
name affiliation country
Tom Birdsong
Kitware, Inc.
USA

Description

The goal of this breakout session is to gather all parties interested in the future of rendering in VTK and Slicer, present ongoing development by Kitware and others and discuss potential future directions and clinical and biomedical needs.

During the Breakout Session

Links and notes are organized at https://hackmd.io/Mq81LxbYTfqrwOBRjxrb6Q. It is a markdown based document we can collaboratively & interactively edit.

Once the breakout session is over, we will contribute the information back to this page. See Notes below.

Topics

  • VTK Evolution
    • Status of the replacement for OpenGL (WebGPU)
    • Integration of VTK and VTK.js (common shaders?)
  • Rendering
    • Global illumination
    • Support for high resolution volumes
    • Support for shared graphics contexts
    • Support for GPU pre-processing of volumes and meshes
    • Creation of an experimental rendering module:
      • Support for multiple volumes/surfaces handled by the same pipeline
      • Modifiable shaders
      • Multipass rendering
      • Arbitrary number of transfer functions
      • Better support for animation and high-resolution rendering
  • AR-VR
    • New Slicer Mixed-Reality module (for Hololens remoting)
    • Support for OpenXR in Slicer Virtual Reality module
    • Status of WebXR in vtk.js

Notes

VTK C++: WebGPU

Contact: Sankhesh Jhaveri @ Kitware

WebGPU effort in VTK aims to provide a future-proof rendering backend as an alternative to OpenGL.

  • VTK’s data model and visualization pipeline will have no architectural changes.
  • For the most part, there will be no frontend user-facing changes to the rendering classes either. Applications would still have to instantiate vtkRenderWindow, vtkRenderer, vtkActor, etc.
  • There will be changes with respect to how platform-specific windows are created. These windows no longer need to be tied to the graphics backend i.e. no need for vtkXOpenGLRenderWindow, etc.
  • For advanced users of VTK who modify rendering logic, use shader replacements, etc. there will be significant changes. This should be expected, IMO.

Experimental:

Questions:

  • multi-volume rendering
    • Will it be the default ? In the OpenGL backend, there were two backends.
    • vtkMultiVolume issues
      • SetVisibility issue
      • UserMatrix not taken into account for the first volume (issue) (works in TestGPURayCastMultiVolumeOverlapping)
      • Cropping
      • Shading
      • Only shows up in Composite w/ Shading, but nothing in MIP (Error: Shader failed to compile in raycastervs.glsl)
  • context sharing ?
    • Goal is to reduce the memory footprint by avoiding having duplicate of teture memory (e.g multiple 3D views of the same volume data)
    • Availability of result of compute shader for rendering.
  • Compute shaders
    • WebGPU ?

Notes:

WebAssembly

Contact: Jaswant Panchumarti @ Kitware

VTK C++: OpenXR

Contact: Lucas Gandel @ Kitware

Improvement roadmap for OpenXR and OpenXR Remoting support in VTK:

  • OpenXR controller model support:
    • vtkOpenXRManager already supports for querying the controller models in a buffer (see here)
    • The buffer should be forwarded to vtkGLTFReader using vtkResourceStream/vtkResourceParser
    • Need to add vtkResourceStream/vtkResourceParser support to vtkGLTFReader (see PLYReader )
  • OpenXR Remoting hand interaction improvements
    • The current interactor style does not allow for accessing different poses from a single Move3D event (see discussions in this MR)
    • Possible options are:
      • Add support for storing additional poses (maybe as additional TrackedDevice)
      • Add support for hands joints tracking and implement basic gestures recognition
  • Add support for scene understanding in OpenXR Remoting, to occlude the VTK scene with the real world.

SlicerVirtualReality

  • Are we ready to transition to OpenXR ?

SlicerMixedReality

Contact: Jean-Christophe Fillion-Robin @ Kitware

See KitwareMedical/SlicerMixedReality#2

Question: What to do once we start working on adding OpenXR support to SlicerVirtualReality.

vtk.js: Rendering Non-uniform image series

Contact: Shreeraj Jadhav @ Kitware

See https://docs.google.com/presentation/d/1mrMe8w2G5hgRan0KzdwqrxgKGLfyR-h3mM7Kj5KYz4c/edit#slide=id.p

vtk.js: Interactive, in-browser cinematic volume rendering of medical images

Contact: Stephen Aylward & Forrest Li @ Kitware

The diversity and utility of cinematic volume rendering (CVR) for medical image visualisation have grown rapidly in recent years. At the same time, volume rendering on augmented and virtual reality systems is attracting greater interest with the advance of the WebXR standard.

See https://doi.org/10.1080/21681163.2022.2145239

See https://volview.kitware.com/

vtk.js: WebXR

Contact: Tom Birdsong @ Kitware

VTK & ITK interoperability

Web:

  • ITK IO are compiled to WASM and re-used in vtk.js based web application
  • ITK-WASM is a building block for VTK-WASM

OME-Zarr:

Slicer + Large Image Rendering:

  • We currently do not have any readily available implementation in Slicer, from a high level, the idea would be to:

    • support streaming dataset at different resolutions and updating the rendering accordingly.
    • leverage ITK+Zarr integration and ensure information flow all the way through the VTK pipeline
      • Consolidate ITKVTKGlue modules

  • Improve ITK module: ITKIOOMEZarrNGFF. See https://github.com/InsightSoftwareConsortium/ITKIOOMEZarrNGFF

Illustrations

Background and References