Extended Reality with HMD-Assisted Guidance and Console 3D Overlay for Robotic Surgery Remote Mentoring

RA-L 2024

Chengkun Li1 · Yuqi Tong1 · Yonghao Long1 · Weixin Si2 · David Chun Man Yeung3 · Jason Ying-Kuen Chan3 · Pheng-Ann Heng1 · Qi Dou1
1Dept. of Computer Science and Engineering, The Chinese University of Hong Kong
2Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences
3Dept. of Otorhinolaryngology, Head and Neck Surgery, The Chinese University of Hong Kong
Paper Code

The prototype of our remote mentoring surgery's extended reality system (RMS-XR).

Abstract

The concept of remote-guided surgery has garnered significant attention among researchers as a possible solution to transcend geographic barriers and facilitate the integration of medical resources across diverse regions. However, establishing effective communication channels between remote specialists and surgeons poses an ongoing challenge in this field. In this paper, we propose an extended reality (XR) remote mentoring system for robotic surgery. The system employs an optical see-through head-mounted display (OST-HMD) to provide real-time 3D endoscopic visualization and enable remote mentors to offer surgical guidance through natural hand gestures and virtual surgical instruments. Additionally, a console 3D overlay method is introduced to enhance the depth perception of the console surgeon by stereo occlusion-aware overlaying augmented surgical tools onto the real endoscopic video. The system incorporates the assistant's role in the surgical team by transmitting the assistant's view to the console surgeon and remote mentor, enriching the information in the operating room. Latency and accuracy experiments have proved the efficiency of the system. The user study in a simulated surgical task shows promising performance for optimizing user experience and reducing communication costs.

Overview

In our system, the computing server directly captures the stereo laparoscopic video from the surgical robot to predict the disparity map. Then, the RGB and disparity maps of the surgical scenario are transmitted to local assistants and remote mentor clients for further reconstruction of the 3D surgical scenario in the XR device. Besides, the remote experts can naturally use hand-based interaction to grasp the virtual instruments to indicate their suggestions, or use the label tool to type and highlight critical anatomy. The guidance from remote mentors is stereo occlusion-aware overlaid on the endoscopic stream and shown in the robot’s console. Additionally, the assistant’s perspective is transmitted to both the console surgeon and the remote mentor, thereby enriching the information in the operating environment.

BibTeX

@ARTICLE{10669229,
  author={Li, Chengkun and Tong, Yuqi and Long, Yonghao and Si, Weixin and Yeung, David Chun Man and Chan, Jason Ying-Kuen and Dou, Qi},
  journal={IEEE Robotics and Automation Letters}, 
  title={Extended Reality With HMD-Assisted Guidance and Console 3D Overlay for Robotic Surgery Remote Mentoring}, 
  year={2024},
  volume={9},
  number={10},
  pages={9135-9142},
  keywords={Surgery;Extended reality;Three-dimensional displays;Mentoring;Instruments;Sockets;Virtual reality;Computer vision;Medical robotics;Virtual reality and interfaces;computer vision for medical robotics;surgical robotics},
  doi={10.1109/LRA.2024.3455936}}