Towards Holistic Surgical Scene Understanding

Abstract

Most benchmarks for studying surgical interventions focus on a specific challenge instead of leveraging the intrinsic complementarity among different tasks. In this work, we present a new experimental framework towards holistic surgical scene understanding. First, we introduce the Phase, Step, Instrument, and Atomic Visual Action recognition (PSI-AVA) Dataset. PSI-AVA includes annotations for both long-term (Phase and Step recognition) and short-term reasoning (Instrument detection and novel Atomic Action recognition) in robot-assisted radical prostatectomy videos. Second, we present Transformers for Action, Phase, Instrument, and steps Recognition (TAPIR) as a strong baseline for surgical scene understanding. TAPIR leverages our dataset's multi-level annotations as it benefits from the learned representation on the instrument detection task to improve its classification capacity. Our experimental results in both PSI-AVA and other publicly available databases demonstrate the adequacy of our framework to spur future research on holistic surgical scene understanding.

MICCAI 2022 Oral. Official extension published at arXiv:2401.11174 . Data and codes available at https://github.com/BCV-Uniandes/TAPIR

References in corpus (2)

• The Kinetics Human Action Video Dataset
• The SARAS Endoscopic Surgeon Action Detection (ESAD) dataset: Challenges and methods

Cited by in corpus (5)

• MATIS: Masked-Attention Transformers for Surgical Instrument Segmentation
• Evaluating the Task Generalization of Temporal Convolutional Networks for Surgical Gesture and Motion Recognition using Kinematic Data
• COMPASS: A Formal Framework and Aggregate Dataset for Generalized Surgical Procedure Modeling
• Exploring the Effect of Dataset Diversity in Self-Supervised Learning for Surgical Computer Vision
• SurgicalGPT: End-to-End Language-Vision GPT for Visual Question Answering in Surgery