AP-MTL: Attention Pruned Multi-task Learning Model for Real-time Instrument Detection and Segmentation in Robot-assisted Surgery

Abstract

Surgical scene understanding and multi-tasking learning are crucial for image-guided robotic surgery. Training a real-time robotic system for the detection and segmentation of high-resolution images provides a challenging problem with the limited computational resource. The perception drawn can be applied in effective real-time feedback, surgical skill assessment, and human-robot collaborative surgeries to enhance surgical outcomes. For this purpose, we develop a novel end-to-end trainable real-time Multi-Task Learning (MTL) model with weight-shared encoder and task-aware detection and segmentation decoders. Optimization of multiple tasks at the same convergence point is vital and presents a complex problem. Thus, we propose an asynchronous task-aware optimization (ATO) technique to calculate task-oriented gradients and train the decoders independently. Moreover, MTL models are always computationally expensive, which hinder real-time applications. To address this challenge, we introduce a global attention dynamic pruning (GADP) by removing less significant and sparse parameters. We further design a skip squeeze and excitation (SE) module, which suppresses weak features, excites significant features and performs dynamic spatial and channel-wise feature re-calibration. Validating on the robotic instrument segmentation dataset of MICCAI endoscopic vision challenge, our model significantly outperforms state-of-the-art segmentation and detection models, including best-performed models in the challenge.

arXiv commentAccepted in the conference of ICRA 2020

References in corpus (18)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Rethinking Atrous Convolution for Semantic Image Segmentation
• YOLOv3: An Incremental Improvement
• Squeeze-and-Excitation Networks
• LinkNet: Exploiting Encoder Representations for Efficient Semantic Segmentation
• Pruning Convolutional Neural Networks for Resource Efficient Inference
• A Survey on Multi-Task Learning
• TernausNet: U-Net with VGG11 Encoder Pre-Trained on ImageNet for Image Segmentation
• Automatic Instrument Segmentation in Robot-Assisted Surgery Using Deep Learning
• Detection and Localization of Robotic Tools in Robot-Assisted Surgery Videos Using Deep Neural Networks for Region Proposal and Detection
• ToolNet: Holistically-Nested Real-Time Segmentation of Robotic Surgical Tools
• Real-Time Instrument Segmentation in Robotic Surgery using Auxiliary Supervised Deep Adversarial Learning
• Skip Connections Eliminate Singularities
• 2017 Robotic Instrument Segmentation Challenge
• Real-time Joint Object Detection and Semantic Segmentation Network for Automated Driving
• Visualizing Uncertainty and Saliency Maps of Deep Convolutional Neural Networks for Medical Imaging Applications
• Learning Where to Look While Tracking Instruments in Robot-assisted Surgery

Cited by in corpus (5)

• Global-Reasoned Multi-Task Learning Model for Surgical Scene Understanding
• One to Many: Adaptive Instrument Segmentation via Meta Learning and Dynamic Online Adaptation in Robotic Surgical Video
• Learning Domain Adaptation with Model Calibration for Surgical Report Generation in Robotic Surgery
• Learning and Reasoning with the Graph Structure Representation in Robotic Surgery
• Real-time landmark detection for precise endoscopic submucosal dissection via shape-aware relation network