5 papers · 1 filter
EasyBalance: Cross-Layer Load Balancing in Distributed MoE Inference
Yize Wu, Ke Gao, Ling Li +1
Load Balancing has emerged as a critical problem in expert-parallel distributed inference of Mixture-of-Experts (MoE) models. As routing distributions are typically skewed across e…
QiMeng-Attention: SOTA Attention Operator is generated by SOTA Attention Algorithm
Qirui Zhou, Shaohui Peng, Weiqiang Xiong +11
The attention operator remains a critical performance bottleneck in large language models (LLMs), particularly for long-context scenarios. While FlashAttention is the most widely u…
QiMeng-TensorOp: Automatically Generating High-Performance Tensor Operators with Hardware Primitives
Xuzhi Zhang, Shaohui Peng, Qirui Zhou +12
Computation-intensive tensor operators constitute over 90\% of the computations in Large Language Models (LLMs) and Deep Neural Networks.Automatically and efficiently generating hi…
Online Pseudo-average Shifting Attention(PASA) for Robust Low-precision LLM Inference: Algorithms and Numerical Analysis
Long Cheng, Qichen Liao, Fan Wu +10
Attention calculation is extremely time-consuming for long-sequence inference tasks, such as text or image/video generation, in large models. To accelerate this process, we develop…
EasySpec: Layer-Parallel Speculative Decoding for Efficient Multi-GPU Utilization
Yize Wu, Ke Gao, Ling Li +1
Speculative decoding is an effective and lossless method for Large Language Model (LLM) inference acceleration. It employs a smaller model to generate a draft token sequence, which…