6 papers
Beyond Objective Expressivity: Geometry Preservation in Multimodal Contrastive Learning
Tillmann Rheude, Roland Eils, Benjamin Wild
Contrastive learning is increasingly moving toward settings with three or more modalities instead of image-text pairs. Yet, extending models from pairwise to higher-order multimoda…
Hidden in the Multiplicative Interaction: Uncovering Fragility in Multimodal Contrastive Learning
Tillmann Rheude, Stefan Hegselmann, Roland Eils +1
Contrastive learning has become a standard approach for unsupervised learning from paired data, as demonstrated by CLIP for image-text matching. However, many domains involve more…
Fusion or Confusion? Multimodal Complexity Is Not All You Need
Tillmann Rheude, Roland Eils, Benjamin Wild
Multimodal learning has become a prominent research area, with the potential of substantial performance gains by combining information across modalities. At the same time, model de…
Cohort-Based Active Modality Acquisition
Tillmann Rheude, Roland Eils, Benjamin Wild
Real-world multimodal machine learning often faces missing, costly-to-acquire modalities, raising the problem of which samples to prioritize for additional acquisition under a budg…
Large Language Models are Powerful Electronic Health Record Encoders
Stefan Hegselmann, Georg von Arnim, Tillmann Rheude +5
Electronic Health Records (EHRs) offer considerable potential for clinical prediction, but their complexity and heterogeneity challenge traditional machine learning. Domain-specifi…
JanusDNA: A Powerful Bi-directional Hybrid DNA Foundation Model
Qihao Duan, Bingding Huang, Zhenqiao Song +4
Large language models (LLMs) have revolutionized natural language processing and are increasingly applied to other sequential data types, including genetic sequences. However, adap…