7 papers
ResMerge: Residual-based Spectral Merging of Large Language Models
Yandu Sun, Zhiyan Hou, Haokai Ma +7
Model merging offers a training-free way to combine multiple post-trained expert models, but merging experts obtained through reinforcement learning (RL) remains challenging. Exist…
Finding the Correct Visual Evidence Without Forgetting: Mitigating Hallucination in LVLMs via Inter-Layer Visual Attention Discrepancy
Yutong Xie, Zhenglin Hua, Ran Wang +3
Large Vision-Language Models (LVLMs) have shown remarkable performance on a wide range of vision-language tasks. Despite this progress, they are still prone to hallucination, gener…
PLUME: Latent Reasoning Based Universal Multimodal Embedding
Chenwei He, Xiangzhao Hao, Tianyu Yang +6
Universal multimodal embedding (UME) maps heterogeneous inputs into a shared retrieval space with a single model. Recent approaches improve UME by generating explicit chain-of-thou…
Rethinking Representativeness and Diversity in Dynamic Data Selection
Yuzhe Zhou, Zhenglin Hua, Haiyun Guo +1
Dynamic data selection accelerates training by sampling a changing subset of the dataset while preserving accuracy. We rethink two core notions underlying sample evaluation: repres…
MLLM-CTBench: A Benchmark for Continual Instruction Tuning with Reasoning Process Diagnosis
Haiyun Guo, Zhiyan Hou, Yandu Sun +6
Continual instruction tuning(CIT) during the post-training phase is crucial for adapting multimodal large language models (MLLMs) to evolving real-world demands. However, the progr…
Steering LVLMs via Sparse Autoencoder for Hallucination Mitigation
Zhenglin Hua, Jinghan He, Zijun Yao +4
Large vision-language models (LVLMs) have achieved remarkable performance on multimodal tasks. However, they still suffer from hallucinations, generating text inconsistent with vis…