1 citations · 1 across the 14 of their papers we have counts for
14 papers
AFD-Ledger: Deployment Provisioning for Attention--FFN Disaggregation
Chengyu Qiu, Xiao Fu, Fengcun Li +6
Attention--Feed-Forward Network (FFN) Disaggregation (AFD) is emerging as a promising architecture for serving Mixture-of-Experts (MoE) language models. While existing AFD systems…
OScaR: The Occam's Razor for Extreme KV Cache Quantization in LLMs and Beyond
Zunhai Su, Rui Yang, Chao Zhang +12
The rapid advancement toward long-context reasoning and multi-modal intelligence has made the memory footprint of the Key-Value (KV) cache a dominant memory bottleneck for efficien…
Attention Sink in Transformers: A Survey on Utilization, Interpretation, and Mitigation
Zunhai Su, Hengyuan Zhang, Wei Wu +24
As the foundational architecture of modern machine learning, Transformers have driven remarkable progress across diverse AI domains. Despite their transformative impact, a persiste…
LongCat-Next: Lexicalizing Modalities as Discrete Tokens
Meituan LongCat Team, Bin Xiao, Chao Wang +86
The prevailing Next-Token Prediction (NTP) paradigm has driven the success of large language models through discrete autoregressive modeling. However, contemporary multimodal syste…
Scaling Embeddings Outperforms Scaling Experts in Language Models
Hong Liu, Jiaqi Zhang, Chao Wang +13
While Mixture-of-Experts (MoE) architectures have become the standard for sparsity scaling in large language models, they increasingly face diminishing returns and system-level bot…
SnapMLA: Efficient Long-Context MLA Decoding via Hardware-Aware FP8 Quantized Pipelining
Yifan Zhang, Zunhai Su, Shuhao Hu +5
While FP8 attention has shown substantial promise in innovations like FlashAttention-3, its integration into the decoding phase of the DeepSeek Multi-head Latent Attention (MLA) ar…
LongCat-Flash-Thinking-2601 Technical Report
Meituan LongCat Team, Anchun Gui, Bei Li +162
We introduce LongCat-Flash-Thinking-2601, a 560-billion-parameter open-source Mixture-of-Experts (MoE) reasoning model with superior agentic reasoning capability. LongCat-Flash-Thi…
Efficient Context Scaling with LongCat ZigZag Attention
Chen Zhang, Yang Bai, Jiahuan Li +19
We introduce LongCat ZigZag Attention (LoZA), which is a sparse attention scheme designed to transform any existing full-attention models into sparse versions with rather limited c…
d3LLM: Ultra-Fast Diffusion LLM using Pseudo-Trajectory Distillation
Yu-Yang Qian, Junda Su, Lanxiang Hu +4
Diffusion large language models (dLLMs) offer capabilities beyond those of autoregressive (AR) LLMs, such as parallel decoding and random-order generation. However, realizing these…
LongCat-Flash Technical Report
Meituan LongCat Team, Bayan, Bei Li +179
We introduce LongCat-Flash, a 560-billion-parameter Mixture-of-Experts (MoE) language model designed for both computational efficiency and advanced agentic capabilities. Stemming f…
Introducing LongCat-Flash-Thinking: A Technical Report
Meituan LongCat Team, Anchun Gui, Bei Li +122
We present LongCat-Flash-Thinking, an efficient 560-billion-parameter open-source Mixture-of-Experts (MoE) reasoning model. Its advanced capabilities are cultivated through a metic…
FlashCommunication V2: Bit Splitting and Spike Reserving for Any Bit Communication
Qingyuan Li, Bo Zhang, Hui Kang +4
Nowadays, communication bottlenecks have emerged as a critical challenge in the distributed training and deployment of large language models (LLMs). This paper introduces FlashComm…
Unveiling Super Experts in Mixture-of-Experts Large Language Models
Zunhai Su, Qingyuan Li, Hao Zhang +6
In this study, we report, for the first time, the discovery and systematic investigation of a distinct subset of experts that play a pivotal role in the MoE LLMs' forward inference…
EPS-MoE: Expert Pipeline Scheduler for Cost-Efficient MoE Inference
Yulei Qian, Fengcun Li, Xiangyang Ji +4
The Mixture-of-Experts (MoE) model has emerged as a prominent architecture in the field of Large Language Models (LLMs), providing a better balance between model performance and co…