6 papers
SharQ: Bridging Activation Sparsity and FP4 Quantization for LLM Inference
Haoqian Meng, Yilun Luo, Yafei Zhao +4
Low-bit floating-point formats and semi-structured sparsity are increasingly supported by modern accelerators, yet combining them for LLM activation compression remains challenging…
Post-Training Quantization of OpenPangu Models for Efficient Deployment on Atlas A2
Yilun Luo, Huaqing Zheng, Haoqian Meng +2
Huawei's openPangu-Embedded-1B and openPangu-Embedded-7B are variants of the openPangu large language model, designed for efficient deployment on Ascend NPUs. The 7B variant suppor…
ARCQuant: Boosting NVFP4 Quantization with Augmented Residual Channels for LLMs
Haoqian Meng, Yilun Luo, Yafei Zhao +3
The emergence of fine-grained numerical formats like NVFP4 presents new opportunities for efficient Large Language Model (LLM) inference. However, it is difficult to adapt existing…
MicroMix: Efficient Mixed-Precision Quantization with Microscaling Formats for Large Language Models
Wenyuan Liu, Haoqian Meng, Yilun Luo +3
Quantization significantly accelerates inference in large language models (LLMs) by replacing original high-precision matrices with low-precision counterparts. Recent advances in w…
CrossQuant: A Post-Training Quantization Method with Smaller Quantization Kernel for Precise Large Language Model Compression
Wenyuan Liu, Xindian Ma, Peng Zhang +1
Post-Training Quantization (PTQ) is an effective technique for compressing Large Language Models (LLMs). While many studies focus on quantizing both weights and activations, it is…
3D-RPE: Enhancing Long-Context Modeling Through 3D Rotary Position Encoding
Xindian Ma, Wenyuan Liu, Peng Zhang +1
Inspired by the Bloch Sphere representation, we propose a novel rotary position encoding on a three-dimensional sphere, named 3D Rotary Position Encoding (3D-RPE). 3D-RPE is an adv…