7 papers
EasyBalance: Cross-Layer Load Balancing in Distributed MoE Inference
Yize Wu, Ke Gao, Ling Li +1
Load Balancing has emerged as a critical problem in expert-parallel distributed inference of Mixture-of-Experts (MoE) models. As routing distributions are typically skewed across e…
RepoOMP: Repository-Aware Hotspot OpenMP Parallelization via Dependency-Aware Context Reduction
Yongjie Qian, Ke Gao, Zhibin Zhang +2
OpenMP parallelization of hotspots in mature repositories remains difficult because loop safety and optimization payoff often depend on non-local evidence. Rule-based tools under-p…
EasySpec: Layer-Parallel Speculative Decoding for Efficient Multi-GPU Utilization
Yize Wu, Ke Gao, Ling Li +1
Speculative decoding is an effective and lossless method for Large Language Model (LLM) inference acceleration. It employs a smaller model to generate a draft token sequence, which…
QiMeng-Kernel: Macro-Thinking Micro-Coding Paradigm for LLM-Based High-Performance GPU Kernel Generation
Xinguo Zhu, Shaohui Peng, Jiaming Guo +10
Developing high-performance GPU kernels is critical for AI and scientific computing, but remains challenging due to its reliance on expert crafting and poor portability. While LLMs…
QiMeng-Attention: SOTA Attention Operator is generated by SOTA Attention Algorithm
Qirui Zhou, Shaohui Peng, Weiqiang Xiong +11
The attention operator remains a critical performance bottleneck in large language models (LLMs), particularly for long-context scenarios. While FlashAttention is the most widely u…
QiMeng-TensorOp: Automatically Generating High-Performance Tensor Operators with Hardware Primitives
Xuzhi Zhang, Shaohui Peng, Qirui Zhou +12
Computation-intensive tensor operators constitute over 90\% of the computations in Large Language Models (LLMs) and Deep Neural Networks.Automatically and efficiently generating hi…