SISOPS

Shadow-Free Projection on Dynamic, Deformable Surfaces in the Operating Room

SmartOT Logo

Funded by
Funded by BMFTR
Grant 16SV9239

SISOPS is a research project funded by the German Federal Ministry of Research, Technology and Space (BMFTR) that aims to develop a projection system suitable for use in the surgical environment. Instead of showing information on separate screens that force the surgeon to divert their eyes from the surgical area, the system will enable the projection of information from various medical devices directly onto the surgical drapes. Some of the challenges we try to solve in this project are the projection onto uneven, dynamic surfaces (drapes), prevention of shadowing artifacts (staff), and touchless interaction. The project partners are Dr. Mach GmbH & Co. KG, inomed Medizintechnik GmbH, KIZMO GmbH, University Clinic for Visceral Surgery at the Pius-Hospital Oldenburg, and Computer Graphics and Virtual Reality Research Lab (CGVR).

SISOPS Conceptual View

Figure 1: Conceptual view visualized in Blender.

SISOPS Conceptual View 2

Figure 2: Prototype under development.

Our work (CGVR) focuses primarily on implementing shadow-free projection using multiple projectors. This involves two particularly interesting challenges: first, geometric correction with the help of multiple depth sensors, where the sensor data must be accurately fused; and second, achieving shadow-free rendering itself. While shadows can be eliminated through the use of multiple projectors, tiny, unavoidable calibration inaccuracies — often less than 2 mm — can still lead to blurring artifacts. Addressing these effects is part of our work on blur mitigation.

Geometric Correction (Rectification)

In dynamic projection mapping, the 3D objects to be projected on are often pre-reconstructed as meshes and tracked relative to camera and projector (cf. Siegl et al., Nomoto et al.). Once tracked, the object can simply be rendered from the projector’s perspective with the desired effect. However, for freely deformable surfaces such as surgical drapes, these approaches cannot be applied directly — real-time surface reconstruction is required.

Since the projection area is often occluded by surgical staff from the projector's perspective, we employ multiple projectors and depth cameras, which in turn requires precise multi-view depth-sensor fusion. State-of-the-art methods typically use voxel-based Truncated Signed Distance Functions (TSDFs) (e.g., Dou et al. (1), Dou et al. (2), Yu et al., Eck et al.), which are computationally demanding and prone to resolution loss due to resampling. Furthermore, when applied to dynamic rather than static scenes, both 3D Gaussian Splatting (3DGS) and Neural Radiance Field (NeRF) approaches fail to achieve real-time reconstruction (at least multiple seconds per frame required) of live data even if real-time rendering is feasible (e.g. Sun et al., Gao et al., Yan et al.); or they exhibit artifacts resulting from noisy depth measurements (e.g., Hu et al.).

We therefore developed BlendPCR, a highly efficient point-cloud rendering technique that fuses multiple depth cameras directly in screen space, preserving each sensor’s full resolution. Through adaptive blending, the individual views merge into a seamless result. This approach can be used not only for geometric correction (rendering from the projector’s viewpoint) but also produces visually convincing outputs in a technically simple way:

BlendPCR Conceptual View

1: Note that both Pointersect and P2ENet are rendered from slightly different perspectives and use slightly different preprossesing filters (in terms of erosion & hole filling). Both renderings are taken from the Supplemental Material of HU Y., GONG R., SUN Q., WANG Y.: Low latency point cloud rendering with learned splatting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops (June 2024), pp. 5752–5761.

Further details on our multi-view point-cloud fusion method can be found in our ICAT-EGVE 2024 paper, and its application to geometric correction is described in our ACM VRST 2025 paper.

Shadow-Avoidance and Blur Mitigation

In a multi-projector setup projecting onto the same surface, even sub-millimeter misalignments between multiple projectors can cause visible blurring, which cannot always be fully avoided technically. To mitigate shadows and blurring, we developed a blur-mitigation pipeline integrated with BlendPCR, which assigns each projected surface region primarily to a single projector.

Pipeline for Blur Mitigation

Figure 3: Blur Mitigation Pipeline, further descripted in our ACM VRST 2025 paper

Our ACM VRST ’25 video illustrates the method and its application in an intuitive way:

Video 1: Presentation of our ACM VRST 2025 paper

Further details are described in our ACM VRST 2025 paper.

Gesture Control

In addition, we are developing the technical implementation of gesture control within the project. This is achieved using MediaPipe for 2D hand detection, followed by projection onto the 3D point cloud to track the hand in 3D space and project it onto a surface. More details will be available soon.

Links

Publications