6 papers
DeaMoE: Efficient MoE Structure for Fast Small-Batch Decoding
Zewen Jin, Shen Fu, Zeping Duan +8
Mixture-of-Experts (MoE) models have been widely adopted in real-time interactive applications such as coding assistants, real-time audio-video interaction systems. To meet the ext…
AdaCluster: Adaptive Query-Key Clustering for Sparse Attention in Video Generation
Haoyue Tan, Shengnan Wang, Yulin Qiao +5
Video diffusion transformers (DiTs) suffer from prohibitive inference latency due to quadratic attention complexity. Existing sparse attention methods either overlook semantic simi…
Lagom: Unleashing the Power of Communication and Computation Overlapping for Distributed LLM Training
Guanbin Xu, ZhenGuo Xu, Yuzhe Li +4
Overlapping communication with computation is crucial for distributed large-model training, yet optimizing it - especially when computation becomes the bottleneck-remains challengi…
LiteCache: A Query Similarity-Driven, GPU-Centric KVCache Subsystem for Efficient LLM Inference
Jiawei Yi, Ping Gong, Youhui Bai +10
During LLM inference, KVCache memory usage grows linearly with sequence length and batch size and often exceeds GPU capacity. Recent proposals offload KV states to host memory and…
HATA: Trainable and Hardware-Efficient Hash-Aware Top-k Attention for Scalable Large Model Inference
Ping Gong, Jiawei Yi, Shengnan Wang +13
Large Language Models (LLMs) have emerged as a pivotal research area, yet the attention module remains a critical bottleneck in LLM inference, even with techniques like KVCache to…
CentroidKV: Efficient Long-Context LLM Inference via KV Cache Clustering
Jie Hu, Shengnan Wang, Yutong He +8
Large language models (LLMs) with extended context windows have become increasingly prevalent for tackling complex tasks. However, the substantial Key-Value (KV) cache required for…