Unsupervised Multimodal Representation Learning across Medical Images and Reports

Abstract

Joint embeddings between medical imaging modalities and associated radiology reports have the potential to offer significant benefits to the clinical community, ranging from cross-domain retrieval to conditional generation of reports to the broader goals of multimodal representation learning. In this work, we establish baseline joint embedding results measured via both local and global retrieval methods on the soon to be released MIMIC-CXR dataset consisting of both chest X-ray images and the associated radiology reports. We examine both supervised and unsupervised methods on this task and show that for document retrieval tasks with the learned representations, only a limited amount of supervision is needed to yield results comparable to those of fully-supervised methods.

arXiv commentMachine Learning for Health (ML4H) Workshop at NeurIPS 2018 arXiv:1811.07216

References in corpus (14)

• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Generative Adversarial Text to Image Synthesis
• NLTK: The Natural Language Toolkit
• CheXNet: Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learning
• Universal Sentence Encoder
• Self-Normalizing Neural Networks
• Word Translation Without Parallel Data
• Unsupervised Cross-Modal Alignment of Speech and Text Embedding Spaces
• Large Scale Automated Reading of Frontal and Lateral Chest X-Rays using Dual Convolutional Neural Networks
• A Universal Music Translation Network
• Unsupervised Alignment of Embeddings with Wasserstein Procrustes
• Learning to Read Chest X-Rays: Recurrent Neural Cascade Model for Automated Image Annotation
• Producing radiologist-quality reports for interpretable artificial intelligence
• Learning to Write Notes in Electronic Health Records

Cited by in corpus (6)

• Multi-modal Understanding and Generation for Medical Images and Text via Vision-Language Pre-Training
• Clinically Accurate Chest X-Ray Report Generation
• Representation Learning for Electronic Health Records
• Mapping individual differences in cortical architecture using multi-view representation learning
• A Comparison of Pre-trained Vision-and-Language Models for Multimodal Representation Learning across Medical Images and Reports
• Unsupervised Clinical Language Translation