TokenSwap: Benchmarking and Reducing the Modality Gap in Multimodal LLMs

Andong Hua, Colton Bishop, Igor Mordatch +5

Multimodal large language models (MLLMs) should generate consistent responses given semantically equivalent inputs across modalities. However, we observe a systematic discrepancy i…