23 papers
AFD-Ledger: Deployment Provisioning for Attention--FFN Disaggregation
Chengyu Qiu, Xiao Fu, Fengcun Li +6
Attention--Feed-Forward Network (FFN) Disaggregation (AFD) is emerging as a promising architecture for serving Mixture-of-Experts (MoE) language models. While existing AFD systems…
LongCat Sparse Attention: Taming the Lightning via Streaming-aware Hierarchical Cross-Layer Indexing
Wen Zan, Jiaqi Zhang, Jianchao Tan +11
DeepSeek Sparse Attention (DSA) enables efficient long-context modeling through its Lightning Indexer. However, practical deployment remains constrained by the indexer's expensive…
Omni-Flow: A Unified Workflow Orchestration and Distributed KV Cache Sharing Framework for Multimodal Inference
Bin Xiao, Jingfu Dong, Changran Wang +5
As large language model (LLM) inference evolves from text-only to multimodal paradigms, inference systems face three challenges: (1) flexible orchestration of multimodal workflows,…
Attention Sink in Transformers: A Survey on Utilization, Interpretation, and Mitigation
Zunhai Su, Hengyuan Zhang, Wei Wu +24
As the foundational architecture of modern machine learning, Transformers have driven remarkable progress across diverse AI domains. Despite their transformative impact, a persiste…
Breaking the Scale Barrier: One-Shot Knowledge Transfer via Frequency Transform
Jianlu Shen, Fu Feng, Yucheng Xie +2
Transferring knowledge by fine-tuning large-scale pre-trained networks has become a standard paradigm for downstream tasks, yet the knowledge of a pre-trained model is tightly coup…
MONA: Muon Optimizer with Nesterov Acceleration for Scalable Language Model Training
Jiacheng Li, Jianchao Tan, Hongtao Xu +5
The Muon optimizer has recently offered a promising alternative to AdamW for large language model training, leveraging matrix orthogonalization to produce geometry-aware updates. H…