Knowledge Matters: Radiology Report Generation with General and Specific Knowledge

Abstract

Automatic radiology report generation is critical in clinics which can relieve experienced radiologists from the heavy workload and remind inexperienced radiologists of misdiagnosis or missed diagnose. Existing approaches mainly formulate radiology report generation as an image captioning task and adopt the encoder-decoder framework. However, in the medical domain, such pure data-driven approaches suffer from the following problems: 1) visual and textual bias problem; 2) lack of expert knowledge. In this paper, we propose a knowledge-enhanced radiology report generation approach introduces two types of medical knowledge: 1) General knowledge, which is input independent and provides the broad knowledge for report generation; 2) Specific knowledge, which is input dependent and provides the fine-grained knowledge for report generation. To fully utilize both the general and specific knowledge, we also propose a knowledge-enhanced multi-head attention mechanism. By merging the visual features of the radiology image with general knowledge and specific knowledge, the proposed model can improve the quality of generated reports. Experimental results on two publicly available datasets IU-Xray and MIMIC-CXR show that the proposed knowledge enhanced approach outperforms state-of-the-art image captioning based methods. Ablation studies also demonstrate that both general and specific knowledge can help to improve the performance of radiology report generation.

arXiv commentMedical Image Analysis

References in corpus (8)

• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Neural Message Passing for Quantum Chemistry
• A review of deep learning in medical imaging: Imaging traits, technology trends, case studies with progress highlights, and future promises
• Publicly Available Clinical BERT Embeddings
• MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographs
• Hybrid Retrieval-Generation Reinforced Agent for Medical Image Report Generation
• Do Transformers Really Perform Bad for Graph Representation?
• Entity, Relation, and Event Extraction with Contextualized Span Representations

Cited by in corpus (6)

• Automated Radiology Report Generation: A Review of Recent Advances
• Cross-Modal Causal Intervention for Medical Report Generation
• Automatic Medical Report Generation: Methods and Applications
• Structural Entities Extraction and Patient Indications Incorporation for Chest X-ray Report Generation
• Factual Serialization Enhancement: A Key Innovation for Chest X-ray Report Generation
• Structure Observation Driven Image-Text Contrastive Learning for Computed Tomography Report Generation