Learning to Read Chest X-Rays: Recurrent Neural Cascade Model for Automated Image Annotation

Abstract

Despite the recent advances in automatically describing image contents, their applications have been mostly limited to image caption datasets containing natural images (e.g., Flickr 30k, MSCOCO). In this paper, we present a deep learning model to efficiently detect a disease from an image and annotate its contexts (e.g., location, severity and the affected organs). We employ a publicly available radiology dataset of chest x-rays and their reports, and use its image annotations to mine disease names to train convolutional neural networks (CNNs). In doing so, we adopt various regularization techniques to circumvent the large normal-vs-diseased cases bias. Recurrent neural networks (RNNs) are then trained to describe the contexts of a detected disease, based on the deep CNN features. Moreover, we introduce a novel approach to use the weights of the already trained pair of CNN/RNN on the domain-specific image/text dataset, to infer the joint image/text contexts for composite image labeling. Significantly improved image annotation results are demonstrated using the recurrent neural cascade model by taking the joint image/text contexts into account.

References in corpus (9)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Scikit-learn: Machine Learning in Python
• Sequence to Sequence Learning with Neural Networks
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Improving neural networks by preventing co-adaptation of feature detectors
• Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
• Visualizing and Understanding Recurrent Networks
• Explain Images with Multimodal Recurrent Neural Networks

Cited by in corpus (8)

• A Survey on Deep Learning in Medical Image Analysis
• Diagnose like a Radiologist: Attention Guided Convolutional Neural Network for Thorax Disease Classification
• Learning to diagnose from scratch by exploiting dependencies among labels
• Learning to detect chest radiographs containing lung nodules using visual attention networks
• XLSor: A Robust and Accurate Lung Segmentor on Chest X-Rays Using Criss-Cross Attention and Customized Radiorealistic Abnormalities Generation
• Lung Segmentation from Chest X-rays using Variational Data Imputation
• Unsupervised Multimodal Representation Learning across Medical Images and Reports
• Chest X-rays Classification: A Multi-Label and Fine-Grained Problem