8 papers
KernelFlume: Elastic Core-Attention Scaling for Agentic Long-Context Decoding
Guangyu Xiang, Xueze Kang, Lin Zhang +4
LLM serving is increasingly dominated by long and dynamic decode workloads from agents, reasoning models, and extended conversations. When bursty long-context demand exceeds deploy…
AGoQ: Activation and Gradient Quantization for Memory-Efficient Distributed Training of LLMs
Wenxiang Lin, Juntao Huang, Luhan Zhang +5
Quantization is a key method for reducing the GPU memory requirement of training large language models (LLMs). Yet, current approaches are ineffective for 4-bit activations and 8-b…
ZipCCL: Efficient Lossless Data Compression of Communication Collectives for Accelerating LLM Training
Wenxiang Lin, Xinglin Pan, Ruibo Fan +2
Communication has emerged as a critical bottleneck in the distributed training of large language models (LLMs). While numerous approaches have been proposed to reduce communication…
Efficient MoE Inference with Fine-Grained Scheduling of Disaggregated Expert Parallelism
Xinglin Pan, Shaohuai Shi, Wenxiang Lin +4
The mixture-of-experts (MoE) architecture scales model size with sublinear computational increase but suffers from memory-intensive inference due to KV caches and sparse expert act…
HierMoE: Accelerating MoE Training with Hierarchical Token Deduplication and Expert Swap
Wenxiang Lin, Xinglin Pan, Lin Zhang +3
The sparsely activated mixture-of-experts (MoE) transformer has become a common architecture for large language models (LLMs) due to its sparsity, which requires fewer computationa…
FSMoE: A Flexible and Scalable Training System for Sparse Mixture-of-Experts Models
Xinglin Pan, Wenxiang Lin, Lin Zhang +5
Recent large language models (LLMs) have tended to leverage sparsity to reduce computations, employing the sparsely activated mixture-of-experts (MoE) technique. MoE introduces fou…