5 papers
INT v.s. FP: A Comprehensive Study of Fine-Grained Low-bit Quantization Formats
Mengzhao Chen, Meng Wu, Hui Jin +10
Modern AI hardware, such as Nvidia's Blackwell architecture, is increasingly embracing low-precision floating-point (FP) formats to handle the pervasive activation outliers in Larg…
GatePro: Parameter-Free Expert Selection Optimization for Mixture-of-Experts Models
Chen Zheng, Yuhang Cai, Deyi Liu +7
Modern large language models leverage Mixture-of-Experts (MoE) architectures for efficient scaling, but face a critical challenge: functionally similar experts are often selected s…
Balanced Actor Initialization: Stable RLHF Training of Distillation-Based Reasoning Models
Chen Zheng, Yiyuan Ma, Yuan Yang +11
The development of alignment and reasoning capabilities in large language models has seen remarkable progress through two paradigms: instruction tuning and reinforcement learning f…
Model Merging in Pre-training of Large Language Models
Yunshui Li, Yiyuan Ma, Shen Yan +23
Model merging has emerged as a promising technique for enhancing large language models, though its application in large-scale pre-training remains relatively unexplored. In this pa…
Scaling Law for Quantization-Aware Training
Mengzhao Chen, Chaoyi Zhang, Jing Liu +8
Large language models (LLMs) demand substantial computational and memory resources, creating deployment challenges. Quantization-aware training (QAT) addresses these challenges by…