4 papers
MoESD: Unveil Speculative Decoding's Potential for Accelerating Sparse MoE
Zongle Huang, Lei Zhu, Zongyuan Zhan +5
Large Language Models (LLMs) have achieved remarkable success across many applications, with Mixture of Experts (MoE) models demonstrating great potential. Compared to traditional…
E-Pruner: Towards Efficient, Economical, and Effective Layer Pruning for Large Language Models
Tao Yuan, Haoli Bai, Yinfei Pan +5
With the increasing size of large language models, layer pruning has gained increased attention as a hardware-friendly approach for model compression. However, existing layer pruni…
HAP: Hybrid Adaptive Parallelism for Efficient Mixture-of-Experts Inference
Haoran Lin, Xianzhi Yu, Kang Zhao +7
Current inference systems for Mixture-of-Experts (MoE) models primarily employ static parallelization strategies. However, these static approaches cannot consistently achieve optim…
FastAttention: Extend FlashAttention2 to NPUs and Low-resource GPUs
Haoran Lin, Xianzhi Yu, Kang Zhao +17
FlashAttention series has been widely applied in the inference of large language models (LLMs). However, FlashAttention series only supports the high-level GPU architectures, e.g.,…