Early Intervention for VFM-based Multimodal Medical Image Classification

Abstract

Current methods for multimodal medical image classification (M3IC) face two major challenges. First, the prevailing "fusion after unimodal image embedding" paradigm cannot fully exploit the complementary and correlated information in the multimodal data. Second, the scarcity of labeled multimodal medical images, coupled with their substantial domain shift from natural images, impedes the use of cutting-edge Vision Foundation Models (VFMs) for medical image embedding. To jointly address the challenges, we propose a novel Early Intervention (EI) framework. Treating one modality as target and the rest as reference, EI harnesses high-level semantic tokens from the reference as intervention tokens to steer the target modality's embedding process at an early stage. For parameter-efficient VFM adaptation, we introduce Mixture of varied-rank LoRAs (MoR), which employs a small set of LoRA experts with distinct ranks and a bypass-allowed router. Extensive experiments on three public datasets, covering retinal disease recognition, skin lesion recognition, and knee anomaly classification, verify the effectiveness of the proposed method against a number of competitive baselines.

Extension of the EI paper (CVPR 2026 findings)