8 papers
CentroidKV: Efficient Long-Context LLM Inference via KV Cache Clustering
Jie Hu, Shengnan Wang, Yutong He +8
Large language models (LLMs) with extended context windows have become increasingly prevalent for tackling complex tasks. However, the substantial Key-Value (KV) cache required for…
HyperParallel-MoE: Multi-Core Interleaved Scheduling for Fast MoE Training on Ascend NPUs
Zewen Jin, Congkun Ai, Guangpeng Zhang +7
Modern Mixture-of-Experts (MoE) models increasingly rely on large-scale AI accelerator clusters for efficient training. Ascend NPUs expose heterogeneous on-chip compute resources,…
Accelerating Long-Tail Generation in Synchronous RLHF Training via Adaptive Tensor Parallelism
Long Zhao, Qinghe Wang, Jiaan Zhu +5
Reinforcement Learning from Human Feedback (RLHF) has become a key post-training paradigm for improving model quality. However, the synchronous three-stage RLHF pipeline is often b…
AdaCluster: Adaptive Query-Key Clustering for Sparse Attention in Video Generation
Haoyue Tan, Shengnan Wang, Yulin Qiao +5
Video diffusion transformers (DiTs) suffer from prohibitive inference latency due to quadratic attention complexity. Existing sparse attention methods either overlook semantic simi…
LiteCache: A Query Similarity-Driven, GPU-Centric KVCache Subsystem for Efficient LLM Inference
Jiawei Yi, Ping Gong, Youhui Bai +10
During LLM inference, KVCache memory usage grows linearly with sequence length and batch size and often exceeds GPU capacity. Recent proposals offload KV states to host memory and…
Lagom: Unleashing the Power of Communication and Computation Overlapping for Distributed LLM Training
Guanbin Xu, ZhenGuo Xu, Yuzhe Li +4
Overlapping communication with computation is crucial for distributed large-model training, yet optimizing it - especially when computation becomes the bottleneck-remains challengi…