2 citations · 2 across the 13 of their papers we have counts for
16 papers · 1 filter
Autonomy-of-Heads: Data-Free Sparse Attention from Frozen Query-Key Geometry
Yehan Yang, Junyuan Shang, Yang Li +3
Long-context LLM inference is bottlenecked by quadratic attention computation and growing KV-cache costs. Existing sparse attention and KV-compression methods typically decide whic…
ConSA: Controllable Sparsity in Hybrid Attention via Learnable Allocation
Yao Chen, Yinqi Yang, Junyuan Shang +6
Hybrid architectures combining full attention (FA) and sliding-window attention (SWA) are a promising paradigm for efficient LLM inference. However, existing methods typically rely…
Elastic MoE: Unlocking the Inference-Time Scalability of Mixture-of-Experts
Naibin Gu, Zhenyu Zhang, Yuchen Feng +8
Mixture-of-Experts (MoE) models typically fix the number of activated experts at both training and inference. However, real-world deployments often face heterogeneous hardware,…
Sparse Growing Transformer: Training-Time Sparse Depth Allocation via Progressive Attention Looping
Yao Chen, Yilong Chen, Yinqi Yang +9
Existing approaches to increasing the effective depth of Transformers predominantly rely on parameter reuse, extending computation through recursive execution. Under this paradigm,…
ERNIE 5.0 Technical Report
Haifeng Wang, Hua Wu, Tian Wu +432
In this report, we introduce ERNIE 5.0, a natively autoregressive foundation model desinged for unified multimodal understanding and generation across text, image, video, and audio…
BeamLoRA: Beam-Constraint Low-Rank Adaptation
Naibin Gu, Zhenyu Zhang, Xiyu Liu +7
Due to the demand for efficient fine-tuning of large language models, Low-Rank Adaptation (LoRA) has been widely adopted as one of the most effective parameter-efficient fine-tunin…