4 papers
HiFA4: Training-Free 4-bit FlashAttention on Ascend HIF4 NPUs for LLM Inference
Hui Dong, Yanzhao Li, Jie Gao +5
We present HiFA4, a post-training operator-level design that executes both QK^T and PV in FlashAttention as 4-bit HIF4 Cube GEMMs for LLM inference on Ascend NPUs, while maintainin…
VFA: Relieving Vector Operations in Flash Attention with Global Maximum Pre-computation
Yupeng Sun, Yanzhao Li, Zhiqiang Zou +5
FlashAttention-style online softmax enables exact attention computation with linear memory by streaming score tiles through on-chip memory and maintaining a running maximum and nor…
OSC: Hardware Efficient W4A4 Quantization via Outlier Separation in Channel Dimension
Zhiyuan Zhang, Yanzhao Li, Zhiqiang Zou +4
While 4-bit quantization is essential for high-throughput deployment of Large Language Models, activation outliers often lead to significant accuracy degradation due to the restric…
E-ConvNeXt: A Lightweight and Efficient ConvNeXt Variant with Cross-Stage Partial Connections
Fang Wang, Huitao Li, Wenhan Chao +4
Many high-performance networks were not designed with lightweight application scenarios in mind from the outset, which has greatly restricted their scope of application. This paper…