Auxiliary Signal-Guided Knowledge Encoder-Decoder for Medical Report Generation
arXiv:2006.03744
Abstract
Beyond the common difficulties faced in the natural image captioning, medical report generation specifically requires the model to describe a medical image with a fine-grained and semantic-coherence paragraph that should satisfy both medical commonsense and logic. Previous works generally extract the global image features and attempt to generate a paragraph that is similar to referenced reports; however, this approach has two limitations. Firstly, the regions of primary interest to radiologists are usually located in a small area of the global image, meaning that the remainder parts of the image could be considered as irrelevant noise in the training procedure. Secondly, there are many similar sentences used in each medical report to describe the normal regions of the image, which causes serious data bias. This deviation is likely to teach models to generate these inessential sentences on a regular basis. To address these problems, we propose an Auxiliary Signal-Guided Knowledge Encoder-Decoder (ASGK) to mimic radiologists' working patterns. In more detail, ASGK integrates internal visual feature fusion and external medical linguistic information to guide medical knowledge transfer and learning. The core structure of ASGK consists of a medical graph encoder and a natural language decoder, inspired by advanced Generative Pre-Training (GPT). Experiments on the CX-CHR dataset and our COVID-19 CT Report dataset demonstrate that our proposed ASGK is able to generate a robust and accurate report, and moreover outperforms state-of-the-art methods on both medical terminology classification and paragraph generation metrics.
11 pages, 3 figures
References in corpus (12)
- Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
- BioBERT: a pre-trained biomedical language representation model for biomedical text mining
- Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
- Focal Loss for Dense Object Detection
- Sequence Level Training with Recurrent Neural Networks
- COVID-CT-Dataset: A CT Scan Dataset about COVID-19
- Improved Image Captioning via Policy Gradient optimization of SPIDEr
- Unsupervised Domain Adaptation through Self-Supervision
- Abnormality Detection and Localization in Chest X-Rays using Deep Convolutional Neural Networks
- Review Networks for Caption Generation
- TieNet: Text-Image Embedding Network for Common Thorax Disease Classification and Reporting in Chest X-rays
- Pretrained Encyclopedia: Weakly Supervised Knowledge-Pretrained Language Model