Gesture Recognition in Robotic Surgery: a Review

Abstract

Objective: Surgical activity recognition is a fundamental step in computer-assisted interventions. This paper reviews the state-of-the-art in methods for automatic recognition of fine-grained gestures in robotic surgery focusing on recent data-driven approaches and outlines the open questions and future research directions. Methods: An article search was performed on 5 bibliographic databases with the following search terms: robotic, robot-assisted, JIGSAWS, surgery, surgical, gesture, fine-grained, surgeme, action, trajectory, segmentation, recognition, parsing. Selected articles were classified based on the level of supervision required for training and divided into different groups representing major frameworks for time series analysis and data modelling. Results: A total of 52 articles were reviewed. The research field is showing rapid expansion, with the majority of articles published in the last 4 years. Deep-learning-based temporal models with discriminative feature extraction and multi-modal data integration have demonstrated promising results on small surgical datasets. Currently, unsupervised methods perform significantly less well than the supervised approaches. Conclusion: The development of large and diverse open-source datasets of annotated demonstrations is essential for development and validation of robust solutions for surgical gesture recognition. While new strategies for discriminative feature extraction and knowledge transfer, or unsupervised and semi-supervised approaches, can mitigate the need for data and labels, they have not yet been demonstrated to achieve comparable performance. Important future research directions include detection and forecast of gesture-specific errors and anomalies. Significance: This paper is a comprehensive and structured analysis of surgical gesture recognition methods aiming to summarize the status of this rapidly evolving field.

arXiv commentin IEEE Transactions on Biomedical Engineering, 2021

References in corpus (13)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Surgical Data Science: Enabling Next-Generation Surgery
• Multi-Level Discovery of Deep Options
• TricorNet: A Hybrid Temporal Convolutional and Recurrent Network for Video Action Segmentation
• Less is More: Surgical Phase Recognition with Less Annotations through Self-Supervised Pre-training of CNN-LSTM Networks
• Learning from a tiny dataset of manual annotations: a teacher/student approach for surgical phase recognition
• Towards Generalizable Surgical Activity Recognition Using Spatial Temporal Graph Convolutional Networks
• Multitask Learning of Temporal Connectionism in Convolutional Networks using a Joint Distribution Loss Function to Simultaneously Identify Tools and Phase in Surgical Videos
• Surgical Gesture Recognition with Optical Flow only
• ESAD: Endoscopic Surgeon Action Detection Dataset
• Be Like Water: Robustness to Extraneous Variables Via Adaptive Feature Normalization
• More unlabelled data or label more data? A study on semi-supervised laparoscopic image segmentation
• DESK: A Robotic Activity Dataset for Dexterous Surgical Skills Transfer to Medical Robots

Cited by in corpus (9)

• Accelerating Surgical Robotics Research: A Review of 10 Years With the da Vinci Research Kit
• Snapture -- A Novel Neural Architecture for Combined Static and Dynamic Hand Gesture Recognition
• Studying the Impact of TensorFlow and PyTorch Bindings on Machine Learning Software Quality
• Think Step by Step: Chain-of-Gesture Prompting for Error Detection in Robotic Surgical Videos
• Evaluating the Task Generalization of Temporal Convolutional Networks for Surgical Gesture and Motion Recognition using Kinematic Data
• SEDMamba: Enhancing Selective State Space Modelling with Bottleneck Mechanism and Fine-to-Coarse Temporal Fusion for Efficient Error Detection in Robot-Assisted Surgery
• COMPASS: A Formal Framework and Aggregate Dataset for Generalized Surgical Procedure Modeling
• Testing Human-Hand Segmentation on In-Distribution and Out-of-Distribution Data in Human-Robot Interactions Using a Deep Ensemble Model
• Neural Network Based Lidar Gesture Recognition for Realtime Robot Teleoperation