11 papers · 1 filter
Self-Distillation for Multi-Token Prediction
Guoliang Zhao, Ruobing Xie, An Wang +3
As Large Language Models (LLMs) scale up, inference efficiency becomes a critical bottleneck. Multi-Token Prediction (MTP) could accelerate LLM inference by predicting multiple fut…
Hunyuan-TurboS: Advancing Large Language Models through Mamba-Transformer Synergy and Adaptive Chain-of-Thought
Tencent Hunyuan Team, Ao Liu, Botong Zhou +248
As Large Language Models (LLMs) rapidly advance, we introduce Hunyuan-TurboS, a novel large hybrid Transformer-Mamba Mixture of Experts (MoE) model. It synergistically combines Mam…
Autonomy-of-Experts Models
Ang Lv, Ruobing Xie, Yining Qian +5
Mixture-of-Experts (MoE) models mostly use a router to assign tokens to specific expert modules, activating only partial parameters and often outperforming dense models. We argue t…
The Climb Carves Wisdom Deeper Than the Summit: On the Noisy Rewards in Learning to Reason
Ang Lv, Ruobing Xie, Xingwu Sun +2
Recent studies on post-training large language models (LLMs) for reasoning through reinforcement learning (RL) typically focus on tasks that can be accurately verified and rewarded…
Magnifier Prompt: Tackling Multimodal Hallucination via Extremely Simple Instructions
Yuhan Fu, Ruobing Xie, Jiazhen Liu +4
Hallucinations in multimodal large language models (MLLMs) hinder their practical applications. To address this, we propose a Magnifier Prompt (MagPrompt), a simple yet effective m…
Language Models "Grok" to Copy
Ang Lv, Ruobing Xie, Xingwu Sun +2
We examine the pre-training dynamics of language models, focusing on their ability to copy text from preceding context--a fundamental skill for various LLM applications, including…