Towards Better Surgical Instrument Segmentation in Endoscopic Vision: Multi-Angle Feature Aggregation and Contour Supervision

Abstract

Accurate and real-time surgical instrument segmentation is important in the endoscopic vision of robot-assisted surgery, and significant challenges are posed by frequent instrument-tissue contacts and continuous change of observation perspective. For these challenging tasks more and more deep neural networks (DNN) models are designed in recent years. We are motivated to propose a general embeddable approach to improve these current DNN segmentation models without increasing the model parameter number. Firstly, observing the limited rotation-invariance performance of DNN, we proposed the Multi-Angle Feature Aggregation (MAFA) method, leveraging active image rotation to gain richer visual cues and make the prediction more robust to instrument orientation changes. Secondly, in the end-to-end training stage, the auxiliary contour supervision is utilized to guide the model to learn the boundary awareness, so that the contour shape of segmentation mask is more precise. The proposed method is validated with ablation experiments on the novel Sinus-Surgery datasets collected from surgeons' operations, and is compared to the existing methods on a public dataset collected with a da Vinci Xi Robot.

arXiv commentAccepted by IEEE Robotics and Automation Letters

References in corpus (10)

• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• LinkNet: Exploiting Encoder Representations for Efficient Semantic Segmentation
• TernausNet: U-Net with VGG11 Encoder Pre-Trained on ImageNet for Image Segmentation
• Automatic Instrument Segmentation in Robot-Assisted Surgery Using Deep Learning
• ToolNet: Holistically-Nested Real-Time Segmentation of Robotic Surgical Tools
• Real-Time Instrument Segmentation in Robotic Surgery using Auxiliary Supervised Deep Adversarial Learning
• 2017 Robotic Instrument Segmentation Challenge
• Towards Robotic Eye Surgery: Marker-free, Online Hand-eye Calibration using Optical Coherence Tomography Images
• Real-Time Semantic Segmentation via Multiply Spatial Fusion Network
• Real-time 3D Tracking of Articulated Tools for Robotic Surgery

Cited by in corpus (7)

• Methods and datasets for segmentation of minimally invasive surgical instruments in endoscopic images and videos: A review of the state of the art
• Multi-frame Feature Aggregation for Real-time Instrument Segmentation in Endoscopic Video
• Multitask Learning in Minimally Invasive Surgical Vision: A Review
• Reducing Annotating Load: Active Learning with Synthetic Images in Surgical Instrument Segmentation
• LC-GAN: Image-to-image Translation Based on Generative Adversarial Network for Endoscopic Images
• Recurrent and Spiking Modeling of Sparse Surgical Kinematics
• Contour Primitive of Interest Extraction Network Based on One-Shot Learning for Object-Agnostic Vision Measurement