CLIP in Medical Imaging: A Survey

Abstract

Contrastive Language-Image Pre-training (CLIP), a simple yet effective pre-training paradigm, successfully introduces text supervision to vision models. It has shown promising results across various tasks due to its generalizability and interpretability. The use of CLIP has recently gained increasing interest in the medical imaging domain, serving as a pre-training paradigm for image-text alignment, or a critical component in diverse clinical tasks. With the aim of facilitating a deeper understanding of this promising direction, this survey offers an in-depth exploration of the CLIP within the domain of medical imaging, regarding both refined CLIP pre-training and CLIP-driven applications. In this paper, we (1) first start with a brief introduction to the fundamentals of CLIP methodology; (2) then investigate the adaptation of CLIP pre-training in the medical imaging domain, focusing on how to optimize CLIP given characteristics of medical images and reports; (3) further explore practical utilization of CLIP pre-trained models in various tasks, including classification, dense prediction, and cross-modal tasks; and (4) finally discuss existing limitations of CLIP in the context of medical imaging, and propose forward-looking directions to address the demands of medical imaging domain. Studies featuring technical and practical value are both investigated. We expect this survey will provide researchers with a holistic understanding of the CLIP paradigm and its potential implications. The project page of this survey can also be found on https://github.com/zhaozh10/Awesome-CLIP-in-Medical-Imaging.

Project page available at https://github.com/zhaozh10/Awesome-CLIP-in-Medical-Imaging

References in corpus (89)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Automated Design of Deep Learning Methods for Biomedical Image Segmentation
• BioBERT: a pre-trained biomedical language representation model for biomedical text mining
• Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
• Bootstrap your own latent: A new approach to self-supervised Learning
• Learning to Prompt for Vision-Language Models
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• A Survey on Explainable Artificial Intelligence (XAI): Towards Medical XAI
• Hierarchical Text-Conditional Image Generation with CLIP Latents
• Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing
• Shortcut Learning in Deep Neural Networks
• TotalSegmentator: robust segmentation of 104 anatomical structures in CT images
• Flamingo: a Visual Language Model for Few-Shot Learning
• The Liver Tumor Segmentation Benchmark (LiTS)
• LAION-5B: An open large-scale dataset for training next generation image-text models
• Barlow Twins: Self-Supervised Learning via Redundancy Reduction
• A large annotated medical image dataset for the development and evaluation of segmentation algorithms
• PadChest: A large chest x-ray image dataset with multi-label annotated reports
• Visual Instruction Tuning
• Reproducible scaling laws for contrastive language-image learning
• Med3D: Transfer Learning for 3D Medical Image Analysis
• The KiTS19 Challenge Data: 300 Kidney Tumor Cases with Clinical Context, CT Semantic Segmentations, and Surgical Outcomes
• CLIP-Driven Universal Model for Organ Segmentation and Tumor Detection
• A Survey on Incorporating Domain Knowledge into Deep Learning for Medical Image Analysis
• MS-Net: Multi-Site Network for Improving Prostate Segmentation with Heterogeneous MRI Data
• Iterative fully convolutional neural networks for automatic vertebra segmentation and identification
• Contrastive learning of global and local features for medical image segmentation with limited annotations
• LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day
• Learning random-walk label propagation for weakly-supervised semantic segmentation
• Multi-modal Understanding and Generation for Medical Images and Text via Vision-Language Pre-Training
• Interactive and Explainable Region-guided Radiology Report Generation
• Fully automatic detection and segmentation of abdominal aortic thrombus in post-operative CTA images using deep convolutional neural networks
• Learning Hierarchical Attention for Weakly-supervised Chest X-Ray Abnormality Localization and Diagnosis
• NegBio: a high-performance tool for negation and uncertainty detection in radiology reports
• Generalized Radiograph Representation Learning via Cross-supervision between Images and Free-text Radiology Reports
• Follow My Eye: Using Gaze to Supervise Computer-Aided Diagnosis
• A Foundation Language-Image Model of the Retina (FLAIR): Encoding Expert Knowledge in Text Supervision
• EVA-CLIP: Improved Training Techniques for CLIP at Scale
• DoctorGLM: Fine-tuning your Chinese Doctor is not a Herculean Task
• Bridging the Gap between Object and Image-level Representations for Open-Vocabulary Detection
• ChatCAD+: Towards a Universal and Reliable Interactive CAD using LLMs
• Capturing human categorization of natural images at scale by combining deep networks and cognitive models
• A Systematic Survey of Prompt Engineering on Vision-Language Foundation Models
• Multi-Granularity Cross-modal Alignment for Generalized Medical Visual Representation Learning
• PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answering
• RoentGen: Vision-Language Foundation Model for Chest X-ray Generation
• REFLACX, a dataset of reports and eye-tracking data for localization of abnormalities in chest x-rays
• Quilt-1M: One Million Image-Text Pairs for Histopathology
• STU-Net: Scalable and Transferable Medical Image Segmentation Models Empowered by Large-Scale Supervised Pre-training
• Multi-task Paired Masking with Alignment Modeling for Medical Vision-Language Pre-training
• Towards long-tailed, multi-label disease classification from chest X-ray: Overview of the CXR-LT challenge
• Generalist Foundation Models from a Multimodal Dataset for 3D Computed Tomography
• DualCoOp: Fast Adaptation to Multi-Label Recognition with Limited Annotations
• Foundational Models in Medical Imaging: A Comprehensive Survey and Future Vision
• Does CLIP Benefit Visual Question Answering in the Medical Domain as Much as it Does in the General Domain?
• Extending Pretrained Segmentation Networks with Additional Anatomical Structures
• AbdomenAtlas-8K: Annotating 8,000 CT Volumes for Multi-Organ Segmentation in Three Weeks
• Multimodal C4: An Open, Billion-scale Corpus of Images Interleaved with Text
• Joint Learning of Localized Representations from Medical Images and Reports
• A ChatGPT Aided Explainable Framework for Zero-Shot Medical Image Diagnosis
• Self-supervised Image-text Pre-training With Mixed Data In Chest X-rays
• EchoPrime: A Multi-Video View-Informed Vision-Language Model for Comprehensive Echocardiography Interpretation
• Focused Decoding Enables 3D Anatomical Detection by Transformers
• Contrastive Adapters for Foundation Model Group Robustness
• Qilin-Med-VL: Towards Chinese Large Vision-Language Model for General Healthcare
• Improving Zero-Shot Generalization for CLIP with Synthesized Prompts
• INSPECT: A Multimodal Dataset for Pulmonary Embolism Diagnosis and Prognosis
• Towards Medical Artificial General Intelligence via Knowledge-Enhanced Multimodal Pretraining
• Robust and Interpretable Medical Image Classifiers via Concept Bottleneck Models
• Learning Multi-modal Representations by Watching Hundreds of Surgical Video Lectures
• Medical Vision Language Pretraining: A survey
• One-shot Localization and Segmentation of Medical Images with Foundation Models
• CheXpert Plus: Augmenting a Large Chest X-ray Dataset with Text Radiology Reports, Patient Demographics and Additional Image Formats
• X-TRA: Improving Chest X-ray Tasks with Cross-Modal Retrieval Augmentation
• GridCLIP: One-Stage Object Detection by Grid-Level CLIP Representation Learning
• Robust CLIP: Unsupervised Adversarial Fine-Tuning of Vision Embeddings for Robust Large Vision-Language Models
• Improving Zero-Shot Detection of Low Prevalence Chest Pathologies using Domain Pre-trained Language Models
• UniBrain: Universal Brain MRI Diagnosis with Hierarchical Knowledge-enhanced Pre-training
• Utilizing Synthetic Data for Medical Vision-Language Pre-training: Bypassing the Need for Real Images
• CT-GLIP: 3D Grounded Language-Image Pretraining with CT Scans and Radiology Reports for Full-Body Scenarios
• Towards Reliable Zero Shot Classification in Self-Supervised Models with Conformal Prediction
• FlexR: Few-shot Classification with Language Embeddings for Structured Reporting of Chest X-rays
• Towards General Text-guided Image Synthesis for Customized Multimodal Brain MRI Generation
• ContextMRI: Enhancing Compressed Sensing MRI through Metadata Conditioning
• The Role of Local Alignment and Uniformity in Image-Text Contrastive Learning on Medical Images
• Few-shot medical image classification with simple shape and texture text descriptors using vision-language models
• Bridged Semantic Alignment for Zero-shot 3D Medical Image Diagnosis
• Tumor Location-weighted MRI-Report Contrastive Learning: A Framework for Improving the Explainability of Pediatric Brain Tumor Diagnosis
• Anatomy-Driven Pathology Detection on Chest X-rays

Cited by in corpus (4)

• Brain-Adapter: Enhancing Neurological Disorder Analysis with Adapter-Tuning Multimodal Large Language Models
• Generalist Models in Medical Image Segmentation: A Survey and Performance Comparison with Task-Specific Approaches
• Point, Detect, Count: Multi-Task Medical Image Understanding with Instruction-Tuned Vision-Language Models
• CytoCLIP: Learning Cytoarchitectural Characteristics in Developing Human Brain Using Contrastive Language Image Pre-Training