Multi-frame Feature Aggregation for Real-time Instrument Segmentation in Endoscopic Video

Abstract

Deep learning-based methods have achieved promising results on surgical instrument segmentation. However, the high computation cost may limit the application of deep models to time-sensitive tasks such as online surgical video analysis for robotic-assisted surgery. Moreover, current methods may still suffer from challenging conditions in surgical images such as various lighting conditions and the presence of blood. We propose a novel Multi-frame Feature Aggregation (MFFA) module to aggregate video frame features temporally and spatially in a recurrent mode. By distributing the computation load of deep feature extraction over sequential frames, we can use a lightweight encoder to reduce the computation costs at each time step. Moreover, public surgical videos usually are not labeled frame by frame, so we develop a method that can randomly synthesize a surgical frame sequence from a single labeled frame to assist network training. We demonstrate that our approach achieves superior performance to corresponding deeper segmentation models on two public surgery datasets.

arXiv commentPublished in IEEE Robotics and Automation Letters (Early Access)

References in corpus (12)

• Adam: A Method for Stochastic Optimization
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Spatio-Temporal Graph Convolutional Networks: A Deep Learning Framework for Traffic Forecasting
• A Critical Review of Recurrent Neural Networks for Sequence Learning
• Enhanced LSTM for Natural Language Inference
• TernausNet: U-Net with VGG11 Encoder Pre-Trained on ImageNet for Image Segmentation
• Automatic Instrument Segmentation in Robot-Assisted Surgery Using Deep Learning
• Video-based surgical skill assessment using 3D convolutional neural networks
• ToolNet: Holistically-Nested Real-Time Segmentation of Robotic Surgical Tools
• 2018 Robotic Scene Segmentation Challenge
• Towards Better Surgical Instrument Segmentation in Endoscopic Vision: Multi-Angle Feature Aggregation and Contour Supervision
• 2017 Robotic Instrument Segmentation Challenge

Cited by in corpus (2)

• Methods and datasets for segmentation of minimally invasive surgical instruments in endoscopic images and videos: A review of the state of the art
• Reducing Annotating Load: Active Learning with Synthetic Images in Surgical Instrument Segmentation