Contrastive Cross-Modal Pre-Training: A General Strategy for Small Sample Medical Imaging

Abstract

A key challenge in training neural networks for a given medical imaging task is often the difficulty of obtaining a sufficient number of manually labeled examples. In contrast, textual imaging reports, which are often readily available in medical records, contain rich but unstructured interpretations written by experts as part of standard clinical practice. We propose using these textual reports as a form of weak supervision to improve the image interpretation performance of a neural network without requiring additional manually labeled examples. We use an image-text matching task to train a feature extractor and then fine-tune it in a transfer learning setting for a supervised task using a small labeled dataset. The end result is a neural network that automatically interprets imagery without requiring textual reports during inference. This approach can be applied to any task for which text-image pairs are readily available. We evaluate our method on three classification tasks and find consistent performance improvements, reducing the need for labeled data by 67%-98%.

arXiv commentThis work is accepted to the IEEE Journal of Biomedical and Health Informatics

References in corpus (5)

• A Survey on Deep Learning in Medical Image Analysis
• FaceNet: A Unified Embedding for Face Recognition and Clustering
• BioBERT: a pre-trained biomedical language representation model for biomedical text mining
• Dynamic Image for 3D MRI Image Alzheimer's Disease Classification
• A deep learning view of the census of galaxy clusters in IllustrisTNG

Cited by in corpus (1)

• Clinically Labeled Contrastive Learning for OCT Biomarker Classification