4 papers
Gestalt: Large Multimodal Interplay Model
Zequn Yang, Yu Miao, Haotian Ni +8
In this paper, we propose Gestalt, a new paradigm of large multimodal model built around multimodal interplay. Despite rapid advances, large multimodal models are reaching a bottle…
Information-Theoretic Decomposition for Multimodal Interaction Learning
Zequn Yang, Yake Wei, Haotian Ni +2
Multimodal learning hinges on capturing redundant, unique, and synergistic information across modalities, which collectively constitute multimodal interactions. A critical yet unde…
MIBench: Evaluating LMMs on Multimodal Interaction
Yu Miao, Zequn Yang, Yake Wei +5
In different multimodal scenarios, it needs to integrate and utilize information across modalities in a specific way based on the demands of the task. Different integration ways be…
RollingQ: Reviving the Cooperation Dynamics in Multimodal Transformer
Haotian Ni, Yake Wei, Hang Liu +4
Multimodal learning faces challenges in effectively fusing information from diverse modalities, especially when modality quality varies across samples. Dynamic fusion strategies, s…