Exploring scalable medical image encoders beyond text supervision

Abstract

Language-supervised pre-training has proven to be a valuable method for extracting semantically meaningful features from images, serving as a foundational element in multimodal systems within the computer vision and medical imaging domains. However, the computed features are limited by the information contained in the text, which is particularly problematic in medical imaging, where the findings described by radiologists focus on specific observations. This challenge is compounded by the scarcity of paired imaging-text data due to concerns over leakage of personal health information. In this work, we fundamentally challenge the prevailing reliance on language supervision for learning general-purpose biomedical imaging encoders. We introduce RAD-DINO, a biomedical image encoder pre-trained solely on unimodal biomedical imaging data that obtains similar or greater performance than state-of-the-art biomedical language-supervised models on a diverse range of benchmarks. Specifically, the quality of learned representations is evaluated on standard imaging tasks (classification and semantic segmentation), and a vision-language alignment task (text report generation from images). To further demonstrate the drawback of language supervision, we show that features from RAD-DINO correlate with other medical records (e.g., sex or age) better than language-supervised models, which are generally not mentioned in radiology reports. Finally, we conduct a series of ablations determining the factors in RAD-DINO's performance; notably, we observe that RAD-DINO's downstream performance scales well with the quantity and diversity of training data, demonstrating that image-only supervision is a scalable approach for training a foundational biomedical image encoder. Model weights of RAD-DINO trained on publicly available datasets are available at https://huggingface.co/microsoft/rad-dino.

References in corpus (29)

• Automated Design of Deep Learning Methods for Biomedical Image Segmentation
• Learning Transferable Visual Models From Natural Language Supervision
• ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases
• Bootstrap your own latent: A new approach to self-supervised Learning
• Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
• Flamingo: a Visual Language Model for Few-Shot Learning
• DINOv2: Learning Robust Visual Features without Supervision
• PadChest: A large chest x-ray image dataset with multi-label annotated reports
• CoCa: Contrastive Captioners are Image-Text Foundation Models
• Deep Learning for Chest X-ray Analysis: A Survey
• SimVLM: Simple Visual Language Model Pretraining with Weak Supervision
• Making the Most of Text Semantics to Improve Biomedical Vision--Language Processing
• LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day
• FILIP: Fine-grained Interactive Language-Image Pre-Training
• Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicine
• Contrastive Masked Autoencoders are Stronger Vision Learners
• Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigm
• Scaling Vision Transformers to 22 Billion Parameters
• BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs
• AI can evolve without labels: self-evolving vision transformer for chest X-ray diagnosis through knowledge distillation
• Deep Learning Discovery of Demographic Biomarkers in Echocardiography
• K-LITE: Learning Transferable Visual Models with External Knowledge
• CheXmask: a large-scale dataset of anatomical segmentation masks for multi-center chest x-ray images
• Med-UniC: Unifying Cross-Lingual Medical Vision-Language Pre-Training by Diminishing Bias
• Supervised Transfer Learning at Scale for Medical Imaging
• Effect of Pre-Training Scale on Intra- and Inter-Domain Full and Few-Shot Transfer Learning for Natural and Medical X-Ray Chest Images
• From CLIP to DINO: Visual Encoders Shout in Multi-modal Large Language Models
• Image Captioners Are Scalable Vision Learners Too
• Consensus, dissensus and synergy between clinicians and specialist foundation models in radiology report generation

Cited by in corpus (5)

• M4CXR: Exploring Multi-task Potentials of Multi-modal Large Language Models for Chest X-ray Interpretation
• Generalist Models in Medical Image Segmentation: A Survey and Performance Comparison with Task-Specific Approaches
• MOTOR: Multimodal Optimal Transport via Grounded Retrieval in Medical Visual Question Answering
• Evaluating Vision Language Model Adaptations for Radiology Report Generation in Low-Resource Languages
• DiCoM -- Diverse Concept Modeling towards Enhancing Generalizability in Chest X-Ray Studies