Hybrid Retrieval-Generation Reinforced Agent for Medical Image Report Generation

Abstract

Generating long and coherent reports to describe medical images poses challenges to bridging visual patterns with informative human linguistic descriptions. We propose a novel Hybrid Retrieval-Generation Reinforced Agent (HRGR-Agent) which reconciles traditional retrieval-based approaches populated with human prior knowledge, with modern learning-based approaches to achieve structured, robust, and diverse report generation. HRGR-Agent employs a hierarchical decision-making procedure. For each sentence, a high-level retrieval policy module chooses to either retrieve a template sentence from an off-the-shelf template database, or invoke a low-level generation module to generate a new sentence. HRGR-Agent is updated via reinforcement learning, guided by sentence-level and word-level rewards. Experiments show that our approach achieves the state-of-the-art results on two medical report datasets, generating well-balanced structured sentences with robust coverage of heterogeneous medical report contents. In addition, our model achieves the highest detection accuracy of medical terminologies, and improved human evaluation performance.

References in corpus (14)

• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• A Deep Reinforced Model for Abstractive Summarization
• Improved Image Captioning via Policy Gradient optimization of SPIDEr
• Image Captioning with Semantic Attention
• Toward Controlled Generation of Text
• From Captions to Visual Concepts and Back
• Recurrent Topic-Transition GAN for Visual Paragraph Generation
• Challenges in Data-to-Document Generation
• Hierarchical Text Generation and Planning for Strategic Dialogue
• Video Captioning via Hierarchical Reinforcement Learning
• Texar: A Modularized, Versatile, and Extensible Toolkit for Text Generation
• End-to-End Video Captioning with Multitask Reinforcement Learning
• Generative Bridging Network in Neural Sequence Prediction

Cited by in corpus (16)

• A Survey on Incorporating Domain Knowledge into Deep Learning for Medical Image Analysis
• Knowledge Matters: Radiology Report Generation with General and Specific Knowledge
• Interactive and Explainable Region-guided Radiology Report Generation
• Clinically Accurate Chest X-Ray Report Generation
• Medical-VLBERT: Medical Visual Language BERT for COVID-19 CT Report Generation With Alternate Learning
• Reinforcement Learning Applications
• RepsNet: Combining Vision with Language for Automated Medical Reports
• Chest ImaGenome Dataset for Clinical Reasoning
• Dynamic Interactive Relation Capturing via Scene Graph Learning for Robotic Surgical Report Generation
• Vispi: Automatic Visual Perception and Interpretation of Chest X-rays
• Texar: A Modularized, Versatile, and Extensible Toolkit for Text Generation
• Automatic Medical Report Generation: Methods and Applications
• Learning Hierarchical Representations of Electronic Health Records for Clinical Outcome Prediction
• What Makes A Good Story? Designing Composite Rewards for Visual Storytelling
• Optimizing the Factual Correctness of a Summary: A Study of Summarizing Radiology Reports
• EMIXER: End-to-end Multimodal X-ray Generation via Self-supervision