5 papers
Recurrent Residual Quantization: A Progressive Multi-Precision Representation for LLMs
Yu Luo, Bo Dong, Wenhua Cheng +1
Serving large language models (LLMs) under diverse deployment constraints requires flexible trade-offs between accuracy, memory footprint, and throughput. However, conventional qua…
SignRoundV2: Toward Closing the Performance Gap in Extremely Low-Bit Post-Training Quantization for LLMs
Wenhua Cheng, Weiwei Zhang, Heng Guo +2
Extremely low-bit quantization is critical for efficiently deploying Large Language Models (LLMs), yet it often leads to severe performance degradation at 2 bits and even at 4 bits…
Faster Inference of LLMs using FP8 on the Intel Gaudi
Joonhyung Lee, Shmulik Markovich-Golan, Daniel Ohayon +9
Low-precision data types are essential in modern neural networks during both training and inference as they enhance throughput and computational capacity by better exploiting avail…
A dynamic parallel method for performance optimization on hybrid CPUs
Luo Yu, Liu Yucheng, Shen Haihao
The AIPC concept is gaining popularity, and more and more hybrid CPUs will be running AI models on client devices. However, the current AI inference framework overlooks the imbalan…
Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs
Wenhua Cheng, Weiwei Zhang, Haihao Shen +4
Large Language Models (LLMs) have demonstrated exceptional proficiency in language-related tasks, but their deployment poses significant challenges due to substantial memory and st…