Clinically Accurate Chest X-Ray Report Generation

Abstract

The automatic generation of radiology reports given medical radiographs has significant potential to operationally and improve clinical patient care. A number of prior works have focused on this problem, employing advanced methods from computer vision and natural language generation to produce readable reports. However, these works often fail to account for the particular nuances of the radiology domain, and, in particular, the critical importance of clinical accuracy in the resulting generated reports. In this work, we present a domain-aware automatic chest X-ray radiology report generation system which first predicts what topics will be discussed in the report, then conditionally generates sentences corresponding to these topics. The resulting system is fine-tuned using reinforcement learning, considering both readability and clinical accuracy, as assessed by the proposed Clinically Coherent Reward. We verify this system on two datasets, Open-I and MIMIC-CXR, and demonstrate that our model offers marked improvements on both language generation metrics and CheXpert assessed accuracy over a variety of competitive baselines.

References in corpus (24)

• Adam: A Method for Stochastic Optimization
• A Survey on Deep Learning in Medical Image Analysis
• Sequence to Sequence Learning with Neural Networks
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• CheXNet: Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learning
• PadChest: A large chest x-ray image dataset with multi-label annotated reports
• DenseNet: Implementing Efficient ConvNet Descriptor Pyramids
• How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation
• Deep Reinforcement Learning for Dialogue Generation
• MaskGAN: Better Text Generation via Filling in the______
• Hybrid Retrieval-Generation Reinforced Agent for Medical Image Report Generation
• An Actor-Critic Algorithm for Sequence Prediction
• Exploring Nearest Neighbor Approaches for Image Captioning
• Adversarial Generation of Natural Language
• Language GANs Falling Short
• Large Scale Automated Reading of Frontal and Lateral Chest X-Rays using Dual Convolutional Neural Networks
• Challenges in Data-to-Document Generation
• Scalable Learning of Non-Decomposable Objectives
• Neural Text Generation: A Practical Guide
• Unsupervised Multimodal Representation Learning across Medical Images and Reports
• Producing radiologist-quality reports for interpretable artificial intelligence
• Re-evaluating Automatic Metrics for Image Captioning
• Unsupervised Category Discovery via Looped Deep Pseudo-Task Optimization Using a Large Scale Radiology Image Database
• Learning to Summarize Radiology Findings

Cited by in corpus (23)

• A Survey on Incorporating Domain Knowledge into Deep Learning for Medical Image Analysis
• Release Strategies and the Social Impacts of Language Models
• Contrastive Learning of Medical Visual Representations from Paired Images and Text
• Multi-modal Understanding and Generation for Medical Images and Text via Vision-Language Pre-Training
• Interactive and Explainable Region-guided Radiology Report Generation
• Rationalization for Explainable NLP: A Survey
• RepsNet: Combining Vision with Language for Automated Medical Reports
• Generating Radiology Reports via Memory-driven Transformer
• Representation Learning for Electronic Health Records
• CheXpert++: Approximating the CheXpert labeler for Speed,Differentiability, and Probabilistic Output
• Contrastive Attention for Automatic Chest X-ray Report Generation
• Improving Factual Completeness and Consistency of Image-to-Text Radiology Report Generation
• Using Radiomics as Prior Knowledge for Thorax Disease Classification and Localization in Chest X-rays
• AlignTransformer: Hierarchical Alignment of Visual Regions and Disease Tags for Medical Report Generation
• Lesion Guided Explainable Few Weak-shot Medical Report Generation
• Optimizing the Factual Correctness of a Summary: A Study of Summarizing Radiology Reports
• Reinforced Medical Report Generation with X-Linear Attention and Repetition Penalty
• EMIXER: End-to-end Multimodal X-ray Generation via Self-supervision
• Let Your Heart Speak in its Mother Tongue: Multilingual Captioning of Cardiac Signals
• RTEX: A novel methodology for Ranking, Tagging, and Explanatory diagnostic captioning of radiography exams
• Self-Guided Multiple Instance Learning for Weakly Supervised Disease Classification and Localization in Chest Radiographs
• CLARA: Clinical Report Auto-completion
• Progressive Transformer-Based Generation of Radiology Reports