9 papers
LoopQuant: Accurate and Efficient Post-Training Quantization for Looped Transformers
Zhixiong Zhao, Zukang Xu, Xing Hu +3
Looped Language Models (LoopLMs) improve parameter efficiency by recursively reusing shared Transformer modules, yet their repeated computation over evolving recurrent states poses…
All for 1-Bit: Towards Genuine 1-Bit Post-Training Quantization for LLMs
Zhixiong Zhao, Zukang Xu, Guangyu Sun +2
Large language models (LLMs) have achieved remarkable progress, yet their massive storage and memory-bandwidth demands still hinder efficient deployment. Weight binarization is a p…
ACE: Adaptive Calibration-Free Expert Skipping for MoE-based LLMs
Zukang Xu, Zhixiong Zhao, Xing Hu +5
Mixture-of-Experts (MoE) architectures provide an efficient paradigm for scaling large language models (LLMs), yet fixed top-k routing activates the same number of expert slots for…
TWLA: Achieving Ternary Weights and Low-Bit Activations for LLMs via Post-Training Quantization
Zhixiong Zhao, Zukang Xu, Zhixuan Chen +3
Large language models (LLMs) exhibit exceptional general language processing capabilities, but their memory and compute costs hinder deployment. Ternarization has emerged as a prom…
BWLA: Breaking the Barrier of W1AX Post-Training Quantization for LLMs
Zhixiong Zhao, Zukang Xu, Dawei Yang
Large language models (LLMs) have driven major progress in NLP, yet their substantial memory and compute demands still hinder practical deployment. Binarization can compress weight…
MoBiE: Efficient Inference of Mixture of Binary Experts under Post-Training Quantization
Zhixiong Zhao, Zukang Xu, Zhixuan Chen +1
Mixture-of-Experts (MoE) based large language models (LLMs) offer strong performance but suffer from high memory and computation costs. Weight binarization provides extreme efficie…
KBVQ-MoE: KLT-guided SVD with Bias-Corrected Vector Quantization for MoE Large Language Models
Zukang Xu, Zhixiong Zhao, Xing Hu +2
Mixture of Experts (MoE) models have achieved great success by significantly improving performance while maintaining computational efficiency through sparse expert activation. Howe…
SpecQuant: Spectral Decomposition and Adaptive Truncation for Ultra-Low-Bit LLMs Quantization
Zhixiong Zhao, Fangxin Liu, Junjie Wang +4
The emergence of accurate open large language models (LLMs) has sparked a push for advanced quantization techniques to enable efficient deployment on end-user devices. In this pape…
QUARK: Quantization-Enabled Circuit Sharing for Transformer Acceleration by Exploiting Common Patterns in Nonlinear Operations
Zhixiong Zhao, Haomin Li, Fangxin Liu +5
Transformer-based models have revolutionized computer vision (CV) and natural language processing (NLP) by achieving state-of-the-art performance across a range of benchmarks. Howe…