2 citations · 2 across the 6 of their papers we have counts for
6 papers
Length Value Model: Scalable Value Pretraining for Token-Level Length Modeling
Zhen Zhang, Changyi Yang, Zijie Xia +11
Tokens are the fundamental units of computation in modern autoregressive models, and generation length directly influences both inference cost and reasoning performance. Despite it…
Towards a Comprehensive Scaling Law of Mixture-of-Experts
Guoliang Zhao, Yuhan Fu, Shuaipeng Li +10
Mixture-of-Experts (MoE) models have become the consensus approach for enabling parameter-efficient scaling and cost-effective deployment in large language models. However, existin…
TransMamba: A Sequence-Level Hybrid Transformer-Mamba Language Model
Yixing Li, Ruobing Xie, Zhen Yang +8
Transformers are the cornerstone of modern large language models, but their quadratic computational complexity limits efficiency in long-sequence processing. Recent advancements in…
Scaling Laws for Floating Point Quantization Training
Xingwu Sun, Shuaipeng Li, Ruobing Xie +13
Low-precision training is considered an effective strategy for reducing both training and downstream inference costs. Previous scaling laws for precision mainly focus on integer qu…
Lossless KV Cache Compression to 2%
Zhen Yang, J. N. Han, Kan Wu +4
Large language models have revolutionized data processing in numerous domains, with their ability to handle extended context reasoning receiving notable recognition. To speed up in…
HMoE: Heterogeneous Mixture of Experts for Language Modeling
An Wang, Xingwu Sun, Ruobing Xie +9
Mixture of Experts (MoE) offers remarkable performance and computational efficiency by selectively activating subsets of model parameters. Traditionally, MoE models use homogeneous…