TokenSwap: Benchmarking and Reducing the Modality Gap in Multimodal LLMs

Andong Hua, Colton Bishop, Igor Mordatch +5

Multimodal large language models (MLLMs) should exhibit cross-modal consistency, generating consistent responses given semantically equivalent inputs across modalities. However, we…