Automatic Instrument Segmentation in Robot-Assisted Surgery Using Deep Learning

Abstract

Semantic segmentation of robotic instruments is an important problem for the robot-assisted surgery. One of the main challenges is to correctly detect an instrument's position for the tracking and pose estimation in the vicinity of surgical scenes. Accurate pixel-wise instrument segmentation is needed to address this challenge. In this paper we describe our winning solution for MICCAI 2017 Endoscopic Vision SubChallenge: Robotic Instrument Segmentation. Our approach demonstrates an improvement over the state-of-the-art results using several novel deep neural network architectures. It addressed the binary segmentation problem, where every pixel in an image is labeled as an instrument or background from the surgery video feed. In addition, we solve a multi-class segmentation problem, where we distinguish different instruments or different parts of an instrument from the background. In this setting, our approach outperforms other methods in every task subcategory for automatic instrument segmentation thereby providing state-of-the-art solution for this problem. The source code for our solution is made publicly available at https://github.com/ternaus/robot-surgery-segmentation

arXiv comment9 pages, 3 figures. arXiv admin note: substantial text overlap with arXiv:1804.08024

References in corpus (8)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• TernausNet: U-Net with VGG11 Encoder Pre-Trained on ImageNet for Image Segmentation
• Deep Convolutional Neural Networks for Breast Cancer Histology Image Analysis
• Deep Learning in Pharmacogenomics: From Gene Regulation to Patient Stratification
• Satellite Imagery Feature Detection using Deep Convolutional Neural Network: A Kaggle Competition
• ToolNet: Holistically-Nested Real-Time Segmentation of Robotic Surgical Tools
• Pediatric Bone Age Assessment Using Deep Convolutional Neural Networks
• Angiodysplasia Detection and Localization Using Deep Convolutional Neural Networks

Cited by in corpus (43)

• Albumentations: fast and flexible image augmentations
• CHAOS Challenge -- Combined (CT-MR) Healthy Abdominal Organ Segmentation
• Generalist Vision Foundation Models for Medical Imaging: A Case Study of Segment Anything Model on Zero-Shot Medical Segmentation
• 2018 Robotic Scene Segmentation Challenge
• Real-Time Instrument Segmentation in Robotic Surgery using Auxiliary Supervised Deep Adversarial Learning
• Angiodysplasia Detection and Localization Using Deep Convolutional Neural Networks
• Towards Better Surgical Instrument Segmentation in Endoscopic Vision: Multi-Angle Feature Aggregation and Contour Supervision
• 2017 Robotic Instrument Segmentation Challenge
• Image Compositing for Segmentation of Surgical Tools without Manual Annotations
• Unsupervised Intra-domain Adaptation for Semantic Segmentation through Self-Supervision
• Global-Reasoned Multi-Task Learning Model for Surgical Scene Understanding
• AP-MTL: Attention Pruned Multi-task Learning Model for Real-time Instrument Detection and Segmentation in Robot-assisted Surgery
• Fixed-Point Convolutional Neural Network for Real-Time Video Processing in FPGA
• MATIS: Masked-Attention Transformers for Surgical Instrument Segmentation
• Multi-frame Feature Aggregation for Real-time Instrument Segmentation in Endoscopic Video
• Using Conditional Generative Adversarial Networks to Reduce the Effects of Latency in Robotic Telesurgery
• Sales forecasting using WaveNet within the framework of the Kaggle competition
• Automatic salt deposits segmentation: A deep learning approach
• Federated Split Vision Transformer for COVID-19 CXR Diagnosis using Task-Agnostic Training
• U-NetPlus: A Modified Encoder-Decoder U-Net Architecture for Semantic and Instance Segmentation of Surgical Instrument
• Safety Filtering While Training: Improving the Performance and Sample Efficiency of Reinforcement Learning Agents
• Studying the Impact of TensorFlow and PyTorch Bindings on Machine Learning Software Quality
• Efficient Global-Local Memory for Real-time Instrument Segmentation of Robotic Surgical Video
• SFSegNet: Parse Freehand Sketches using Deep Fully Convolutional Networks
• Semantics-Aware Image to Image Translation and Domain Transfer
• One to Many: Adaptive Instrument Segmentation via Meta Learning and Dynamic Online Adaptation in Robotic Surgical Video
• Towards Deep Robot Learning with Optimizer applicable to Non-stationary Problems
• Multimodal Noisy Segmentation based fragmented burn scars identification in Amazon Rainforest
• LC-GAN: Image-to-image Translation Based on Generative Adversarial Network for Endoscopic Images
• Few-Shot Defect Segmentation Leveraging Abundant Normal Training Samples Through Normal Background Regularization and Crop-and-Paste Operation
• How can we learn (more) from challenges? A statistical approach to driving future algorithm development
• Incorporating Temporal Prior from Motion Flow for Instrument Segmentation in Minimally Invasive Surgery Video
• Farmland Parcel Delineation Using Spatio-temporal Convolutional Networks
• Efficient Multi-robot Exploration via Multi-head Attention-based Cooperation Strategy
• Unsupervised Surgical Instrument Segmentation via Anchor Generation and Semantic Diffusion
• Learning Motion Flows for Semi-supervised Instrument Segmentation from Robotic Surgical Video
• Reachability Analysis of Convolutional Neural Networks
• Learning Where to Look While Tracking Instruments in Robot-assisted Surgery
• Camera Model Identification Using Convolutional Neural Networks
• MaskPlus: Improving Mask Generation for Instance Segmentation
• Unsupervised Adaptive Semantic Segmentation with Local Lipschitz Constraint
• An Abstraction Model for Semantic Segmentation Algorithms
• A Novel Deep ML Architecture by Integrating Visual Simultaneous Localization and Mapping (vSLAM) into Mask R-CNN for Real-time Surgical Video Analysis