SuPer: A Surgical Perception Framework for Endoscopic Tissue Manipulation with Surgical Robotics

Abstract

Traditional control and task automation have been successfully demonstrated in a variety of structured, controlled environments through the use of highly specialized modeled robotic systems in conjunction with multiple sensors. However, the application of autonomy in endoscopic surgery is very challenging, particularly in soft tissue work, due to the lack of high-quality images and the unpredictable, constantly deforming environment. In this work, we propose a novel surgical perception framework, SuPer, for surgical robotic control. This framework continuously collects 3D geometric information that allows for mapping a deformable surgical field while tracking rigid instruments within the field. To achieve this, a model-based tracker is employed to localize the surgical tool with a kinematic prior in conjunction with a model-free tracker to reconstruct the deformable environment and provide an estimated point cloud as a mapping of the environment. The proposed framework was implemented on the da Vinci Surgical System in real-time with an end-effector controller where the target configurations are set and regulated through the framework. Our proposed framework successfully completed soft tissue manipulation tasks with high accuracy. The demonstration of this novel framework is promising for the future of surgical autonomy. In addition, we provide our dataset for further surgical research.

The first two authors made equal contribution on this paper

References in corpus (2)

• Dynamic Reconstruction of Deformable Soft-tissue with Stereo Scope in Minimal Invasive Surgery
• Real-time 3D Tracking of Articulated Tools for Robotic Surgery

Cited by in corpus (8)

• Accelerating Surgical Robotics Research: A Review of 10 Years With the da Vinci Research Kit
• Tracking and Mapping in Medical Computer Vision: A Review
• Pose Estimation for Robot Manipulators via Keypoint Optimization and Sim-to-Real Transfer
• Sim-to-Real Surgical Robot Learning and Autonomous Planning for Internal Tissue Points Manipulation using Reinforcement Learning
• Surgical Tattoos in Infrared: A Dataset for Quantifying Tissue Tracking and Mapping
• Automatic Tissue Traction Using Miniature Force-Sensing Forceps for Minimally Invasive Surgery
• SuPerPM: A Surgical Perception Framework Based on Deep Point Matching Learned from Physical Constrained Simulation Data
• Point Cloud Segmentation for Autonomous Clip Positioning in Laparoscopic Cholecystectomy on a Phantom