To Align or Not to Align: Strategic Multimodal Representation Alignment for Optimal Performance

Wanlong Fang, Tianle Zhang, Alvin Chan

Multimodal learning often relies on aligning representations across modalities to enable effective information integration, an approach traditionally assumed to be universally bene…