ToolNet: Holistically-Nested Real-Time Segmentation of Robotic Surgical Tools

Abstract

Real-time tool segmentation from endoscopic videos is an essential part of many computer-assisted robotic surgical systems and of critical importance in robotic surgical data science. We propose two novel deep learning architectures for automatic segmentation of non-rigid surgical instruments. Both methods take advantage of automated deep-learning-based multi-scale feature extraction while trying to maintain an accurate segmentation quality at all resolutions. The two proposed methods encode the multi-scale constraint inside the network architecture. The first proposed architecture enforces it by cascaded aggregation of predictions and the second proposed network does it by means of a holistically-nested architecture where the loss at each scale is taken into account for the optimization process. As the proposed methods are for real-time semantic labeling, both present a reduced number of parameters. We propose the use of parametric rectified linear units for semantic labeling in these small architectures to increase the regularization ability of the design and maintain the segmentation accuracy without overfitting the training sets. We compare the proposed architectures against state-of-the-art fully convolutional networks. We validate our methods using existing benchmark datasets, including ex vivo cases with phantom tissue and different robotic surgical instruments present in the scene. Our results show a statistically significant improved Dice Similarity Coefficient over previous instrument segmentation methods. We analyze our design choices and discuss the key drivers for improving accuracy.

Paper accepted at IROS 2017

References in corpus (2)

• Conditional Random Fields as Recurrent Neural Networks
• Real-Time Segmentation of Non-Rigid Surgical Tools based on Deep Learning and Tracking

Cited by in corpus (13)

• Weakly-Supervised Convolutional Neural Networks for Multimodal Image Registration
• Automatic Instrument Segmentation in Robot-Assisted Surgery Using Deep Learning
• Accelerating Surgical Robotics Research: A Review of 10 Years With the da Vinci Research Kit
• CAI4CAI: The Rise of Contextual Artificial Intelligence in Computer Assisted Interventions
• Real-Time Instrument Segmentation in Robotic Surgery using Auxiliary Supervised Deep Adversarial Learning
• Generalised Wasserstein Dice Score for Imbalanced Multi-class Segmentation using Holistic Convolutional Networks
• Towards Better Surgical Instrument Segmentation in Endoscopic Vision: Multi-Angle Feature Aggregation and Contour Supervision
• Global-Reasoned Multi-Task Learning Model for Surgical Scene Understanding
• AP-MTL: Attention Pruned Multi-task Learning Model for Real-time Instrument Detection and Segmentation in Robot-assisted Surgery
• Multi-frame Feature Aggregation for Real-time Instrument Segmentation in Endoscopic Video
• Efficient Global-Local Memory for Real-time Instrument Segmentation of Robotic Surgical Video
• Deep Homography Estimation in Dynamic Surgical Scenes for Laparoscopic Camera Motion Extraction
• ROBUST-MIPS: A Combined Skeletal Pose and Instance Segmentation Dataset for Laparoscopic Surgical Instruments