4 papers
CANN Bench: Benchmarking Agent Generated Kernels against Real NPU and Algorithmic Limits
Xue-Jian Gao, Deng Pan, Yueming Su +12
AI agents are now capable of writing, compiling, and iteratively optimizing low-level operator kernels on different hardware platforms. Existing benchmarks, however, focus almost e…
Multi-Scale Dequant: Eliminating Dequantization Bottleneck via Activation Decomposition for Efficient LLM Inference
Lingchao Zheng, Yuwei Fan, Jun Li +5
Quantization is essential for efficient large language model (LLM) inference, yet the dequantization step-converting low-bit weights back to high-precision for matrix multiplicatio…
AMLA: MUL by ADD in FlashAttention Rescaling
Qichen Liao, Chengqiu Hu, Fangzheng Miao +8
Multi-head Latent Attention (MLA) significantly reduces KVCache memory usage in Large Language Models while introducing substantial computational overhead and intermediate variable…
Online Pseudo-average Shifting Attention(PASA) for Robust Low-precision LLM Inference: Algorithms and Numerical Analysis
Long Cheng, Qichen Liao, Fan Wu +10
Attention calculation is extremely time-consuming for long-sequence inference tasks, such as text or image/video generation, in large models. To accelerate this process, we develop…