2 citations · 2 across the 12 of their papers we have counts for
12 papers · 1 filter
Loop the Loopies!
Zitian Gao, Yilong Chen, Yihao Xiao +4
We present the Loopie series, consisting of two Mixture-of-Experts (MoE) models: a 20B-parameter model with 2B active parameters and a 6B-parameter model with 0.6B active parameter…
ConSA: Controllable Sparsity in Hybrid Attention via Learnable Allocation
Yao Chen, Yinqi Yang, Junyuan Shang +6
Hybrid architectures combining full attention (FA) and sliding-window attention (SWA) are a promising paradigm for efficient LLM inference. However, existing methods typically rely…
Elastic MoE: Unlocking the Inference-Time Scalability of Mixture-of-Experts
Naibin Gu, Zhenyu Zhang, Yuchen Feng +8
Mixture-of-Experts (MoE) models typically fix the number of activated experts at both training and inference. However, real-world deployments often face heterogeneous hardware,…
Beyond N-gram: Data-Aware X-GRAM Extraction for Efficient Embedding Parameter Scaling
Yilong Chen, Yanxi Xie, Zitian Gao +10
Large token-indexed lookup tables provide a compute-decoupled scaling path, but their practical gains are often limited by poor parameter efficiency and rapid memory growth. We att…
Sparse Growing Transformer: Training-Time Sparse Depth Allocation via Progressive Attention Looping
Yao Chen, Yilong Chen, Yinqi Yang +9
Existing approaches to increasing the effective depth of Transformers predominantly rely on parameter reuse, extending computation through recursive execution. Under this paradigm,…
ERNIE 5.0 Technical Report
Haifeng Wang, Hua Wu, Tian Wu +432
In this report, we introduce ERNIE 5.0, a natively autoregressive foundation model desinged for unified multimodal understanding and generation across text, image, video, and audio…