32 papers
Hybrid Gated Attention
Zekun Zhou, Ruobing Xie, Lanrui Wang +1
Gated attention is an effective approach to mitigate attention sinks and enhance the representational capacity of attention. To further extend its effectiveness-efficiency Pareto f…
AngelSpec: Towards Real-World High Performance Inference with Speculative Decoding
Hong Liu, Rui Cen, Junhan Shi +10
Speculative decoding accelerates large language model inference without changing the target distribution, but no single drafting structure performs best across real-world workloads…
FastMix: Fast Data Mixture Optimization via Gradient Descent
Haoru Tan, Sitong Wu, Yanfeng Chen +5
While large and diverse datasets have driven recent advances in large models, identifying the optimal data mixture for pre-training and post-training remains a significant open pro…
Beyond Encoder Accumulation: Measuring Encoder Roles in Multi-Encoder VLMs
Wei Ding, Yudong Zhang, Ruobing Xie +3
As foundation models scale toward fusing more heterogeneous visual streams, understanding how diverse encoders interact under joint training becomes a prerequisite for principled d…
Negative Advantages Is a Double-Edged Sword: Calibrating advantages in GRPO for Search Agents
Jiayi Wu, Ruobing Xie, Zeqian Huang +6
Search agents achieve strong question-answering performance through multi-turn interactions with search engines, with Group Relative Policy Optimization (GRPO) being a widely used…
MHSA: A Lightweight Framework for Mitigating Hallucinations via Steered Attention in LVLMs
Wei Ding, Yilin Li, Yudong Zhang +4
Large vision-language models (LVLMs) have achieved remarkable performance across diverse multimodal tasks, yet they continue to suffer from hallucinations, generating content that…