Global-Reasoned Multi-Task Learning Model for Surgical Scene Understanding

Abstract

Global and local relational reasoning enable scene understanding models to perform human-like scene analysis and understanding. Scene understanding enables better semantic segmentation and object-to-object interaction detection. In the medical domain, a robust surgical scene understanding model allows the automation of surgical skill evaluation, real-time monitoring of surgeon's performance and post-surgical analysis. This paper introduces a globally-reasoned multi-task surgical scene understanding model capable of performing instrument segmentation and tool-tissue interaction detection. Here, we incorporate global relational reasoning in the latent interaction space and introduce multi-scale local (neighborhood) reasoning in the coordinate space to improve segmentation. Utilizing the multi-task model setup, the performance of the visual-semantic graph attention network in interaction detection is further enhanced through global reasoning. The global interaction space features from the segmentation module are introduced into the graph network, allowing it to detect interactions based on both node-to-node and global interaction reasoning. Our model reduces the computation cost compared to running two independent single-task models by sharing common modules, which is indispensable for practical applications. Using a sequential optimization technique, the proposed multi-task model outperforms other state-of-the-art single-task models on the MICCAI endoscopic vision challenge 2018 dataset. Additionally, we also observe the performance of the multi-task model when trained using the knowledge distillation technique. The official code implementation is made available in GitHub.

arXiv commentCode available at: https://github.com/lalithjets/Global-reasoned-multi-task-model

References in corpus (11)

• Adam: A Method for Stochastic Optimization
• LinkNet: Exploiting Encoder Representations for Efficient Semantic Segmentation
• When Does Label Smoothing Help?
• GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks
• Mask R-CNN
• Automatic Instrument Segmentation in Robot-Assisted Surgery Using Deep Learning
• ToolNet: Holistically-Nested Real-Time Segmentation of Robotic Surgical Tools
• Hierarchical Graph Pooling with Structure Learning
• 2018 Robotic Scene Segmentation Challenge
• AP-MTL: Attention Pruned Multi-task Learning Model for Real-time Instrument Detection and Segmentation in Robot-assisted Surgery
• Visual-Semantic Graph Attention Networks for Human-Object Interaction Detection

Cited by in corpus (5)

• Encoding Surgical Videos as Latent Spatiotemporal Graphs for Object and Anatomy-Driven Reasoning
• Multitask Learning in Minimally Invasive Surgical Vision: A Review
• SegMatch: A semi-supervised learning method for surgical instrument segmentation
• Surgical-VQA: Visual Question Answering in Surgical Scenes using Transformer
• Latent Graph Representations for Critical View of Safety Assessment