On the Automatic Generation of Medical Imaging Reports

Abstract

Medical imaging is widely used in clinical practice for diagnosis and treatment. Report-writing can be error-prone for unexperienced physicians, and time- consuming and tedious for experienced physicians. To address these issues, we study the automatic generation of medical imaging reports. This task presents several challenges. First, a complete report contains multiple heterogeneous forms of information, including findings and tags. Second, abnormal regions in medical images are difficult to identify. Third, the re- ports are typically long, containing multiple sentences. To cope with these challenges, we (1) build a multi-task learning framework which jointly performs the pre- diction of tags and the generation of para- graphs, (2) propose a co-attention mechanism to localize regions containing abnormalities and generate narrations for them, (3) develop a hierarchical LSTM model to generate long paragraphs. We demonstrate the effectiveness of the proposed methods on two publicly available datasets.

ACL 2018

Cited by in corpus (16)

• An overview of deep learning in medical imaging focusing on MRI
• Generative Adversarial Network in Medical Imaging: A Review
• A review of deep learning in medical imaging: Imaging traits, technology trends, case studies with progress highlights, and future promises
• A General Survey on Attention Mechanisms in Deep Learning
• A Survey on Incorporating Domain Knowledge into Deep Learning for Medical Image Analysis
• BiomedGPT: A Generalist Vision-Language Foundation Model for Diverse Biomedical Tasks
• Interactive and Explainable Region-guided Radiology Report Generation
• Medical-VLBERT: Medical Visual Language BERT for COVID-19 CT Report Generation With Alternate Learning
• Automated Radiology Report Generation: A Review of Recent Advances
• CAMANet: Class Activation Map Guided Attention Network for Radiology Report Generation
• Benchmark datasets driving artificial intelligence development fail to capture the needs of medical professionals
• Dynamic Interactive Relation Capturing via Scene Graph Learning for Robotic Surgical Report Generation
• Automated Enriched Medical Concept Generation for Chest X-ray Images
• Improving 3D convolutional neural network comprehensibility via interactive visualization of relevance maps: Evaluation in Alzheimer's disease
• MvKeTR: Chest CT Report Generation with Multi-View Perception and Knowledge Enhancement
• GIT-CXR: End-to-End Transformer for Chest X-Ray Report Generation