6 papers · 1 filter
Optimizing Long-context LLM Serving via Fine-grained Sequence Parallelism
Cong Li, Yuzhe Yang, Xuegui Zheng +7
With the advancement of large language models (LLMs), their context windows have rapidly expanded. To meet diverse demands from varying-length requests in online services, existing…
LiquidGEMM: Hardware-Efficient W4A8 GEMM Kernel for High-Performance LLM Serving
Huanqi Hu, Bowen Xiao, Shixuan Sun +8
Quantization is a critical technique for accelerating LLM inference by reducing memory footprint and improving computational efficiency. Among various schemes, 4-bit weight and 8-b…
SwiftSpec: Ultra-Low Latency LLM Decoding by Scaling Asynchronous Speculative Decoding
Ziyi Zhang, Ziheng Jiang, Chengquan Jiang +5
Low-latency decoding for large language models (LLMs) is crucial for applications like chatbots and code assistants, yet generating long outputs remains slow in single-query settin…
Triton-distributed: Programming Overlapping Kernels on Distributed AI Systems with the Triton Compiler
Size Zheng, Wenlei Bao, Qi Hou +19
In this report, we propose Triton-distributed, an extension of existing Triton compiler, to overcome the programming challenges in distributed AI systems. Triton-distributed is the…
TileLink: Generating Efficient Compute-Communication Overlapping Kernels using Tile-Centric Primitives
Size Zheng, Jin Fang, Xuegui Zheng +9
Large deep learning models have achieved state-of-the-art performance in a wide range of tasks. These models often necessitate distributed systems for efficient training and infere…
Comet: Fine-grained Computation-communication Overlapping for Mixture-of-Experts
Shulai Zhang, Ningxin Zheng, Haibin Lin +9
Mixture-of-experts (MoE) has been extensively employed to scale large language models to trillion-plus parameters while maintaining a fixed computational cost. The development of l…