Towards Cardiac MRI Foundation Models: Comprehensive Visual-Tabular Representations for Whole-Heart Assessment and Beyond

Abstract

Cardiac magnetic resonance imaging is the gold standard for non-invasive cardiac assessment, offering rich spatio-temporal views of the cardiac anatomy and physiology. Patient-level health factors, such as demographics, metabolic, and lifestyle, are known to substantially influence cardiovascular health and disease risk, yet remain uncaptured by CMR alone. To holistically understand cardiac health and to enable the best possible interpretation of an individual's disease risk, CMR and patient-level factors must be jointly exploited within an integrated framework. Recent multi-modal approaches have begun to bridge this gap, yet they often rely on limited spatio-temporal data and focus on isolated clinical tasks, thereby hindering the development of a comprehensive representation for cardiac health evaluation. To overcome these limitations, we introduce ViTa, a step toward foundation models that delivers a comprehensive representation of the heart and a precise interpretation of individual disease risk. Leveraging data from 42,000 UK Biobank participants, ViTa integrates 3D+T cine stacks from short-axis and long-axis views, enabling a complete capture of the cardiac cycle. These imaging data are then fused with detailed tabular patient-level factors, enabling context-aware insights. This multi-modal paradigm supports a wide spectrum of downstream tasks, including cardiac phenotype and physiological feature prediction, segmentation, and classification of cardiac and metabolic diseases within a single unified framework. By learning a shared latent representation that bridges rich imaging features and patient context, ViTa moves beyond traditional, task-specific models toward a universal, patient-specific understanding of cardiac health, highlighting its potential to advance clinical utility and scalability in cardiac analysis.

References in corpus (14)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Automated Design of Deep Learning Methods for Biomedical Image Segmentation
• A Simple Framework for Contrastive Learning of Visual Representations
• Deep Neural Networks and Tabular Data: A Survey
• Deep learning for cardiac image segmentation: A review
• BiomedGPT: A Generalist Vision-Language Foundation Model for Diverse Biomedical Tasks
• Learning Shape Priors for Robust Cardiac MR Segmentation from Multi-view Images
• Assessment of Deep Learning Segmentation for Real-Time Free-Breathing Cardiac Magnetic Resonance Imaging at Rest and Under Exercise Stress
• Attention-aware non-rigid image registration for accelerated MR imaging
• MedVersa: A Generalist Foundation Model for Medical Image Interpretation
• A Generalizable Deep Learning System for Cardiac MRI
• Direct Cardiac Segmentation from Undersampled K-space Using Transformers
• Whole Heart 3D+T Representation Learning Through Sparse 2D Cardiac MR Images
• Multimodal Representation Learning of Cardiovascular Magnetic Resonance Imaging

Cited by in corpus (1)

• A Generalizable Deep Learning System for Cardiac MRI