6 papers
MF-QAT: Multi-Format Quantization-Aware Training for Elastic Inference
Zifei Xu, Sayeh Sharify, Hesham Mostafa
Quantization-aware training (QAT) is typically performed for a single target numeric format, while practical deployments often need to choose numerical precision at inference time…
CRoPE: Efficient Parametrization of Rotary Positional Embedding
Beicheng Lou, Zifei Xu, Vivian W. H. Wong
Rotary positional embedding has become the state-of-the-art approach to encode position information in transformer-based models. While it is often succinctly expressed in complex l…
The Impact of Quantization on Large Reasoning Model Reinforcement Learning
Medha Kumar, Zifei Xu, Xin Wang +1
Strong reasoning capabilities can now be achieved by large-scale reinforcement learning (RL) without any supervised fine-tuning. Although post-training quantization (PTQ) and quant…
Early Attentive Sparsification Accelerates Neural Speech Transcription
Zifei Xu, Sayeh Sharify, Hesham Mostafa +3
Transformer-based neural speech processing has achieved state-of-the-art performance. Since speech audio signals are known to be highly compressible, here we seek to accelerate neu…
Understanding the Difficulty of Low-Precision Post-Training Quantization for LLMs
Zifei Xu, Sayeh Sharify, Wanzin Yazar +2
Large language models of high parameter counts are computationally expensive, yet can be made much more efficient by compressing their weights to very low numerical precision. This…
Scaling Laws for Post Training Quantized Large Language Models
Zifei Xu, Alexander Lan, Wanzin Yazar +3
Generalization abilities of well-trained large language models (LLMs) are known to scale predictably as a function of model size. In contrast to the existence of practical scaling…