Surgical Instruction Generation with Transformers

Abstract

Automatic surgical instruction generation is a prerequisite towards intra-operative context-aware surgical assistance. However, generating instructions from surgical scenes is challenging, as it requires jointly understanding the surgical activity of current view and modelling relationships between visual information and textual description. Inspired by the neural machine translation and imaging captioning tasks in open domain, we introduce a transformer-backboned encoder-decoder network with self-critical reinforcement learning to generate instructions from surgical images. We evaluate the effectiveness of our method on DAISI dataset, which includes 290 procedures from various medical disciplines. Our approach outperforms the existing baseline over all caption evaluation metrics. The results demonstrate the benefits of the encoder-decoder structure backboned by transformer in handling multimodal context.

arXiv commentAccepted to MICCAI 2021

References in corpus (8)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• On the difficulty of training Recurrent Neural Networks
• Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
• PadChest: A large chest x-ray image dataset with multi-label annotated reports
• Generating Radiology Reports via Memory-driven Transformer
• DAISI: Database for AI Surgical Instruction