Methods and datasets for segmentation of minimally invasive surgical instruments in endoscopic images and videos: A review of the state of the art

Abstract

In the field of computer- and robot-assisted minimally invasive surgery, enormous progress has been made in recent years based on the recognition of surgical instruments in endoscopic images and videos. In particular, the determination of the position and type of instruments is of great interest. Current work involves both spatial and temporal information, with the idea that predicting the movement of surgical tools over time may improve the quality of the final segmentations. The provision of publicly available datasets has recently encouraged the development of new methods, mainly based on deep learning. In this review, we identify and characterize datasets used for method development and evaluation and quantify their frequency of use in the literature. We further present an overview of the current state of research regarding the segmentation and tracking of minimally invasive surgical instruments in endoscopic images and videos. The paper focuses on methods that work purely visually, without markers of any kind attached to the instruments, considering both single-frame semantic and instance segmentation approaches, as well as those that incorporate temporal information. The publications analyzed were identified through the platforms Google Scholar, Web of Science, and PubMed. The search terms used were "instrument segmentation", "instrument tracking", "surgical tool segmentation", and "surgical tool tracking", resulting in a total of 741 articles published between 01/2015 and 07/2023, of which 123 were included using systematic selection criteria. A discussion of the reviewed literature is provided, highlighting existing shortcomings and emphasizing the available potential for future developments.

arXiv comment30 pages, 10 figures

References in corpus (11)

• 2018 Robotic Scene Segmentation Challenge
• Real-Time Instrument Segmentation in Robotic Surgery using Auxiliary Supervised Deep Adversarial Learning
• Real-Time Segmentation of Non-Rigid Surgical Tools based on Deep Learning and Tracking
• Towards Better Surgical Instrument Segmentation in Endoscopic Vision: Multi-Angle Feature Aggregation and Contour Supervision
• Image Compositing for Segmentation of Surgical Tools without Manual Annotations
• Robotic Endoscope Control via Autonomous Instrument Tracking
• Comparative evaluation of instrument segmentation and tracking methods in minimally invasive surgery
• CholecSeg8k: A Semantic Segmentation Dataset for Laparoscopic Cholecystectomy Based on Cholec80
• Multi-frame Feature Aggregation for Real-time Instrument Segmentation in Endoscopic Video
• Assessing YOLACT++ for real time and robust instance segmentation of medical instruments in endoscopic procedures
• Polyp and Surgical Instrument Segmentation with Double Encoder-Decoder Networks

Cited by in corpus (4)

• SurgiTrack: Fine-Grained Multi-Class Multi-Tool Tracking in Surgical Videos
• Comparative validation of surgical phase recognition, instrument keypoint estimation, and instrument instance segmentation in endoscopy: Results of the PhaKIR 2024 challenge
• Leveraging Transfer Learning with Class-Specific Decoders for Laparoscopic Segmentation
• First Investigation of Deep Learning for Intraoperative Gauze Segmentation in Minimally Invasive Abdominal Surgery