7 papers
Beyond Objective Expressivity: Geometry Preservation in Multimodal Contrastive Learning
Tillmann Rheude, Roland Eils, Benjamin Wild
Contrastive learning is increasingly moving toward settings with three or more modalities instead of image-text pairs. Yet, extending models from pairwise to higher-order multimoda…
Modeling Local, Global, and Cross-Modal Context in Multimodal 3D MRI
Minh Duc Do, Tillmann Rheude, Noel Kronenberg +2
Brain MRI poses a fundamental challenge for machine learning: models must learn from high-dimensional 3D data spanning multiple co-registered modalities, despite the limited sample…
Hidden in the Multiplicative Interaction: Uncovering Fragility in Multimodal Contrastive Learning
Tillmann Rheude, Stefan Hegselmann, Roland Eils +1
Contrastive learning has become a standard approach for unsupervised learning from paired data, as demonstrated by CLIP for image-text matching. However, many domains involve more…
Fusion or Confusion? Multimodal Complexity Is Not All You Need
Tillmann Rheude, Roland Eils, Benjamin Wild
Multimodal learning has become a prominent research area, with the potential of substantial performance gains by combining information across modalities. At the same time, model de…
Cohort-Based Active Modality Acquisition
Tillmann Rheude, Roland Eils, Benjamin Wild
Real-world multimodal machine learning often faces missing, costly-to-acquire modalities, raising the problem of which samples to prioritize for additional acquisition under a budg…
Large Language Models are Powerful Electronic Health Record Encoders
Stefan Hegselmann, Georg von Arnim, Tillmann Rheude +5
Electronic Health Records (EHRs) offer considerable potential for clinical prediction, but their complexity and heterogeneity challenge traditional machine learning. Domain-specifi…