Multimodal AI in Healthcare: Review of Vision-Language Foundation Models for Real-World Medical Applications
arXiv:2611.19958 · doi:10.1016/j.jbi.2026.105075
Abstract
The emergence of foundation models has marked a transformative shift in AI, enabling robust generalization across diverse downstream tasks through putative zero-shot learning. Large Language Models and Vision Language Models have demonstrated strong capabilities in tasks such as image interpretation, report generation, and question answering by effectively learning from multimodal data-images paired with associated text often with minimal supervision. In the healthcare domain, this ability to align visual and textual information reduces the reliance on extensive manual annotations, as models can leverage existing clinical reports and imaging data to learn meaningful representations. This integration holds promise for improving diagnostic support, treatment planning, and overall patient care, even in data constrained settings. In this review, we provide a definitive taxonomy of the medical VLM landscape, tracing the evolution from early Contrastive Alignment and Generative MLLMs to the cutting edge frontiers of Dense Pixel Grounding, Sparse Mixture-of-Experts (MoE), and Reasoning Incentivized (RL) architectures. We critically examine the "medical bottleneck" identifying the persistent challenges of data scarcity, the "hallucination" risks in generative diagnostics, the computational strain of 3D volumetric processing, and the lack of standardized, clinically grounded evaluation metrics.
arXiv comment20 Pages, 3 figures, 3 Tables
References in corpus (22)
- Learning Transferable Visual Models From Natural Language Supervision
- Deep Residual Learning for Image Recognition
- Segment Anything in Medical Images
- Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing
- COVID-CT-Dataset: A CT Scan Dataset about COVID-19
- SimVLM: Simple Visual Language Model Pretraining with Weak Supervision
- Augmented CycleGAN: Learning Many-to-Many Mappings from Unpaired Data
- Towards Data-Efficient Learning: A Benchmark for COVID-19 CT Lung and Infection Segmentation
- BiomedGPT: A Generalist Vision-Language Foundation Model for Diverse Biomedical Tasks
- MIMIC-Extract: A Data Extraction, Preprocessing, and Representation Pipeline for MIMIC-III
- 3DSAM-adapter: Holistic adaptation of SAM from 2D to 3D for promptable tumor segmentation
- BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs
- A Foundation Language-Image Model of the Retina (FLAIR): Encoding Expert Knowledge in Text Supervision
- PathVQA: 30000+ Questions for Medical Visual Question Answering
- RoentGen: Vision-Language Foundation Model for Chest X-ray Generation
- Med-Flamingo: a Multimodal Medical Few-shot Learner
- Foundational Models in Medical Imaging: A Comprehensive Survey and Future Vision
- Towards a clinically accessible radiology foundation model: open-access and lightweight, with automated evaluation
- Vision Language Models for Dynamic Human Activity Recognition in Healthcare Settings
- Exact Matching: Algorithms and Related Problems
- COVIDx CXR-4: An Expanded Multi-Institutional Open-Source Benchmark Dataset for Chest X-ray Image-Based Computer-Aided COVID-19 Diagnostics
- Open-Qwen2VL: Compute-Efficient Pre-Training of Fully-Open Multimodal LLMs on Academic Resources