9 papers
Simply Stabilizing the Loop via Fully Looped Transformer
Rao Fu, Zixuan Yang, Jiankun Zhang +4
Scaling model performance typically requires increasing model size. Looped Transformer offers a compelling alternative by iteratively reusing the same Transformer blocks, trading a…
Pause or Fabricate? Training Language Models for Grounded Reasoning
Yiwen Qiu, Linjuan Wu, Yizhou Liu +9
Large language models have achieved remarkable progress on complex reasoning tasks. However, they often implicitly fabricate information when inputs are incomplete, producing confi…
Ranking Reasoning LLMs under Test-Time Scaling
Mohsen Hariri, Michael Hinczewski, Jing Ma +1
Test-time scaling evaluates reasoning LLMs by sampling multiple outputs per prompt, but ranking models in this regime remains underexplored. We formalize dense benchmark ranking un…
Coupling Experts and Routers in Mixture-of-Experts via an Auxiliary Loss
Ang Lv, Jin Ma, Yiyuan Ma +1
Mixture-of-Experts (MoE) models lack explicit constraints to ensure the router's decisions align well with the experts' capabilities, which ultimately limits model performance. To…
INT v.s. FP: A Comprehensive Study of Fine-Grained Low-bit Quantization Formats
Mengzhao Chen, Meng Wu, Hui Jin +10
Modern AI hardware, such as Nvidia's Blackwell architecture, is increasingly embracing low-precision floating-point (FP) formats to handle the pervasive activation outliers in Larg…
GatePro: Parameter-Free Expert Selection Optimization for Mixture-of-Experts Models
Chen Zheng, Yuhang Cai, Deyi Liu +7
Modern large language models leverage Mixture-of-Experts (MoE) architectures for efficient scaling, but face a critical challenge: functionally similar experts are often selected s…