Less is More: Surgical Phase Recognition with Less Annotations through Self-Supervised Pre-training of CNN-LSTM Networks

Abstract

Real-time algorithms for automatically recognizing surgical phases are needed to develop systems that can provide assistance to surgeons, enable better management of operating room (OR) resources and consequently improve safety within the OR. State-of-the-art surgical phase recognition algorithms using laparoscopic videos are based on fully supervised training. This limits their potential for widespread application, since creation of manual annotations is an expensive process considering the numerous types of existing surgeries and the vast amount of laparoscopic videos available. In this work, we propose a new self-supervised pre-training approach based on the prediction of remaining surgery duration (RSD) from laparoscopic videos. The RSD prediction task is used to pre-train a convolutional neural network (CNN) and long short-term memory (LSTM) network in an end-to-end manner. Our proposed approach utilizes all available data and reduces the reliance on annotated data, thereby facilitating the scaling up of surgical phase recognition algorithms to different kinds of surgeries. Additionally, we present EndoN2N, an end-to-end trained CNN-LSTM model for surgical phase recognition and evaluate the performance of our approach on a dataset of 120 Cholecystectomy laparoscopic videos (Cholec120). This work also presents the first systematic study of self-supervised pre-training approaches to understand the amount of annotations required for surgical phase recognition. Interestingly, the proposed RSD pre-training approach leads to performance improvement even when all the training data is manually annotated and outperforms the single pre-training approach for surgical phase recognition presently published in the literature. It is also observed that end-to-end training of CNN-LSTM networks boosts surgical phase recognition performance.

arXiv comment15 pages, 22 figures

References in corpus (6)

• Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting
• Unsupervised Learning of Video Representations using LSTMs
• Deep Predictive Coding Networks for Video Prediction and Unsupervised Learning
• Unsupervised Visual Representation Learning by Context Prediction
• Learning Representations for Automatic Colorization
• M2CAI Workflow Challenge: Convolutional Neural Networks with Time Smoothing and Hidden Markov Model for Video Frames Classification

Cited by in corpus (13)

• TeCNO: Surgical Phase Recognition with Multi-Stage Temporal Convolutional Networks
• Gesture Recognition in Robotic Surgery: a Review
• RSDNet: Learning to Predict Remaining Surgery Duration from Laparoscopic Videos Without Manual Annotations
• OperA: Attention-Regularized Transformers for Surgical Phase Recognition
• Learning from a tiny dataset of manual annotations: a teacher/student approach for surgical phase recognition
• Self-Supervised Learning for Data Scarcity in a Fatigue Damage Prognostic Problem
• Weakly Supervised Temporal Convolutional Networks for Fine-grained Surgical Activity Recognition
• Temporal Segmentation of Surgical Sub-tasks through Deep Learning with Multiple Data Sources
• Multi-Task Recurrent Convolutional Network with Correlation Loss for Surgical Video Analysis
• LRTD: Long-Range Temporal Dependency based Active Learning for Surgical Workflow Recognition
• Temporal Memory Relation Network for Workflow Recognition from Surgical Video
• Data Transformation Insights in Self-supervision with Clustering Tasks
• Unsupervised Temporal Video Segmentation as an Auxiliary Task for Predicting the Remaining Surgery Duration