CholecTriplet2021: A benchmark challenge for surgical action triplet recognition

Abstract

Context-aware decision support in the operating room can foster surgical safety and efficiency by leveraging real-time feedback from surgical workflow analysis. Most existing works recognize surgical activities at a coarse-grained level, such as phases, steps or events, leaving out fine-grained interaction details about the surgical activity; yet those are needed for more helpful AI assistance in the operating room. Recognizing surgical actions as triplets of <instrument, verb, target> combination delivers comprehensive details about the activities taking place in surgical videos. This paper presents CholecTriplet2021: an endoscopic vision challenge organized at MICCAI 2021 for the recognition of surgical action triplets in laparoscopic videos. The challenge granted private access to the large-scale CholecT50 dataset, which is annotated with action triplet information. In this paper, we present the challenge setup and assessment of the state-of-the-art deep learning methods proposed by the participants during the challenge. A total of 4 baseline methods from the challenge organizers and 19 new deep learning algorithms by competing teams are presented to recognize surgical action triplets directly from surgical videos, achieving mean average precision (mAP) ranging from 4.2% to 38.1%. This study also analyzes the significance of the results obtained by the presented approaches, performs a thorough methodological comparison between them, in-depth result analysis, and proposes a novel ensemble method for enhanced recognition. Our analysis shows that surgical workflow analysis is not yet solved, and also highlights interesting directions for future research on fine-grained surgical activity recognition which is of utmost importance for the development of AI in surgery.

arXiv commentCholecTriplet2021 challenge report. Paper accepted at Elsevier journal of Medical Image Analysis. 22 pages, 8 figures, 11 tables. Challenge website: https://cholectriplet2021.grand-challenge.org

References in corpus (13)

• Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• Surgical Data Science: Enabling Next-Generation Surgery
• The Devil is in the Channels: Mutual-Channel Loss for Fine-Grained Image Classification
• Rendezvous: Attention Mechanisms for the Recognition of Surgical Action Triplets in Endoscopic Videos
• Weakly Supervised Convolutional LSTM Approach for Tool Tracking in Laparoscopic Videos
• CAI4CAI: The Rise of Contextual Artificial Intelligence in Computer Assisted Interventions
• The TUM LapChole dataset for the M2CAI 2016 workflow challenge
• Learning from a tiny dataset of manual annotations: a teacher/student approach for surgical phase recognition
• The SARAS Endoscopic Surgeon Action Detection (ESAD) dataset: Challenges and methods
• Multitask Learning of Temporal Connectionism in Convolutional Networks using a Joint Distribution Loss Function to Simultaneously Identify Tools and Phase in Surgical Videos
• Data Splits and Metrics for Method Benchmarking on Surgical Action Triplet Datasets
• Comparative Validation of Machine Learning Algorithms for Surgical Workflow and Skill Analysis with the HeiChole Benchmark

Cited by in corpus (8)

• Comparison of fine-tuning strategies for transfer learning in medical image classification
• CholecTriplet2022: Show me a tool and tell me the triplet -- an endoscopic vision challenge for surgical action triplet detection
• Rendezvous in Time: An Attention-based Temporal Fusion approach for Surgical Triplet Recognition
• Encoding Surgical Videos as Latent Spatiotemporal Graphs for Object and Anatomy-Driven Reasoning
• Data Splits and Metrics for Method Benchmarking on Surgical Action Triplet Datasets
• Evaluating the Task Generalization of Temporal Convolutional Networks for Surgical Gesture and Motion Recognition using Kinematic Data
• Surgical Tattoos in Infrared: A Dataset for Quantifying Tissue Tracking and Mapping
• Latent Graph Representations for Critical View of Safety Assessment