System Feedback in Gesture-Based Interaction in the Operating Room

This thesis evaluates the optimal system feedback for touchless gesture control in the operating room (OR), through a two-part study with non-surgeons and medical professionals.

Description

Sterility remains the most significant barrier to direct digital interaction in the operating room. Traditional input devices require physical contact, leading to time-consuming re-disinfection or inefficient verbal communication with assistants. To overcome these constraints, the SISOPS project implements gesture-based control using ceiling-mounted depth cameras and projectors that display a user interface directly onto the surgical drape. This setup allows surgeons to maintain focus on the surgical wound and interact with the interface without breaking sterility by using a 'double-click' gesture, performed by pinching the thumb and index finger twice.

However, effective touchless interaction relies heavily on good User Experience (UX) design, specifically the sense of control provided by system feedback. While previous studies have shown that multimodal feedback can improve efficiency, they often rely only on time-based metrics in controlled environments, neglecting the high cognitive load and environmental stressors.

To evaluate the system feedback under realistic conditions, a two-part study was conducted with 25 non-experts and 10 surgeons. Participants performed a Stroop test using gesture control while simultaneously completing a LEGO-sorting task to simulate surgical cognitive load, all under simulated OR lighting and noise levels of 65–70 dB. In addition to visual and auditory feedback, a custom-made vibration wristband provided haptic feedback to replicate the tactile 'click' missing in touchless interactions.

The first study compared five feedback modalities: none, visual, visual-auditory, visual-haptic, and trimodal (visual-auditory-haptic), while the second study, involving the surgeons, evaluated four modalities, excluding the ‘no feedback’ condition. Following each condition, the NASA Task Load Index (NASA-TLX) was conducted to assess perceived workload as well as two individual questions regarding pleasantness and input confirmation. Additionally, participants provided a final ranking of all modalities, accompanied by qualitative feedback to explain their preferences.

Results

The evaluation proved that feedback is mandatory. The "no feedback" condition was rejected by all participants and resulted in nearly double the input attempts.

The NASA-TLX scores showed significant differences between ‘no feedback’ and all other conditions (p < 0.005).

Non-surgeons sorted an average of three fewer bricks in the ‘no feedback’ condition, suggesting that the uncertainty regarding input recognition hindered their transition back to the LEGO-sorting task. Both groups achieved their highest sorting rates under the visual-haptic condition, indicating that this modality promotes the fastest return to the second task.

While non-experts preferred trimodal feedback for maximum security, surgeons favored the visual-haptic condition as it provided a reliable "input confirmation" without the distraction of auditory signals that often got lost in OR noise.

Files

Full version of the bachelor's thesis (German only)

License

This original work is copyright by University of Bremen.
Any software of this work is covered by the European Union Public Licence v1.2. To view a copy of this license, visit eur-lex.europa.eu.
The Thesis provided above (as PDF file) is licensed under Attribution-NonCommercial-NoDerivatives 4.0 International.
Any other assets (3D models, movies, documents, etc.) are covered by the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License. To view a copy of this license, visit creativecommons.org.
If you use any of the assets or software to produce a publication, then you must give credit and put a reference in your publication.
If you would like to use our software in proprietary software, you can obtain an exception from the above license (aka. dual licensing). Please contact zach at cs.uni-bremen dot de.