9 papers · 1 filter
Mechanistic Insights into Functional Sparsity in Multimodal LLMs via CoRe Heads
Ruoxi Sun, Quantong Qiu, Juntao Li +3
While Multimodal Large Language Models (MLLMs) demonstrate remarkable proficiency on complex vision-language tasks, the mechanisms by which they extract query-relevant visual featu…
LongRM: Revealing and Unlocking the Context Boundary of Reward Modeling
Zecheng Tang, Baibei Ji, Quantong Qiu +4
Reward model (RM) plays a pivotal role in aligning large language model (LLM) with human preferences. As real-world applications increasingly involve long history trajectories, e.g…
Revisiting Long-context Modeling from Context Denoising Perspective
Zecheng Tang, Baibei Ji, Juntao Li +3
Long-context models (LCMs) have demonstrated great potential in processing long sequences, facilitating many real-world applications. The success of LCMs can be attributed to their…
Unlocking Recursive Thinking of LLMs: Alignment via Refinement
Haoke Zhang, Xiaobo Liang, Cunxiang Wang +2
The OpenAI o1-series models have demonstrated that leveraging long-form Chain of Thought (CoT) can substantially enhance performance. However, the recursive thinking capabilities o…
Unleashing LLM Reasoning Capability via Scalable Question Synthesis from Scratch
Yuyang Ding, Xinyu Shi, Xiaobo Liang +4
Improving the mathematical reasoning capabilities of Large Language Models (LLMs) is critical for advancing artificial intelligence. However, access to extensive, diverse, and high…
Revealing and Mitigating Over-Attention in Knowledge Editing
Pinzheng Wang, Zecheng Tang, Keyan Zhou +3
Large Language Models have demonstrated superior performance across a wide range of tasks, but they still exhibit undesirable errors due to incorrect knowledge learned from the tra…