Lesion Guided Explainable Few Weak-shot Medical Report Generation

Abstract

Medical images are widely used in clinical practice for diagnosis. Automatically generating interpretable medical reports can reduce radiologists' burden and facilitate timely care. However, most existing approaches to automatic report generation require sufficient labeled data for training. In addition, the learned model can only generate reports for the training classes, lacking the ability to adapt to previously unseen novel diseases. To this end, we propose a lesion guided explainable few weak-shot medical report generation framework that learns correlation between seen and novel classes through visual and semantic feature alignment, aiming to generate medical reports for diseases not observed in training. It integrates a lesion-centric feature extractor and a Transformer-based report generation module. Concretely, the lesion-centric feature extractor detects the abnormal regions and learns correlations between seen and novel classes with multi-view (visual and lexical) embeddings. Then, features of the detected regions and corresponding embeddings are concatenated as multi-view input to the report generation module for explainable report generation, including text descriptions and corresponding abnormal regions detected in the images. We conduct experiments on FFA-IR, a dataset providing explainable annotations, showing that our framework outperforms others on report generation for novel diseases.

References in corpus (8)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• BioBERT: a pre-trained biomedical language representation model for biomedical text mining
• MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographs
• Clinically Accurate Chest X-Ray Report Generation
• Show, Describe and Conclude: On Exploiting the Structure Information of Chest X-Ray Reports
• Zero-Shot Detection
• Weak-shot Fine-grained Classification via Similarity Transfer
• Weak-shot Semantic Segmentation by Transferring Semantic Affinity and Boundary

Cited by in corpus (1)

• Cross-Modal Causal Intervention for Medical Report Generation