27 citations · 48 across the 14 of their papers we have counts for
14 papers
Recurrent Residual Quantization: A Progressive Multi-Precision Representation for LLMs
Yu Luo, Bo Dong, Wenhua Cheng +1
Serving large language models (LLMs) under diverse deployment constraints requires flexible trade-offs between accuracy, memory footprint, and throughput. However, conventional qua…
SignRoundV2: Toward Closing the Performance Gap in Extremely Low-Bit Post-Training Quantization for LLMs
Wenhua Cheng, Weiwei Zhang, Heng Guo +2
Extremely low-bit quantization is critical for efficiently deploying Large Language Models (LLMs), yet it often leads to severe performance degradation at 2 bits and even at 4 bits…
Faster Inference of LLMs using FP8 on the Intel Gaudi
Joonhyung Lee, Shmulik Markovich-Golan, Daniel Ohayon +9
Low-precision data types are essential in modern neural networks during both training and inference as they enhance throughput and computational capacity by better exploiting avail…
A dynamic parallel method for performance optimization on hybrid CPUs
Luo Yu, Liu Yucheng, Shen Haihao
The AIPC concept is gaining popularity, and more and more hybrid CPUs will be running AI models on client devices. However, the current AI inference framework overlooks the imbalan…
Efficient LLM Inference on CPUs
Haihao Shen, Hanwen Chang, Bo Dong +2
Large language models (LLMs) have demonstrated remarkable performance and tremendous potential across a wide range of tasks. However, deploying these models has been challenging du…
Effective Quantization for Diffusion Models on CPUs
Hanwen Chang, Haihao Shen, Yiyang Cai +7
Diffusion models have gained popularity for generating images from textual descriptions. Nonetheless, the substantial need for computational resources continues to present a notewo…
TEQ: Trainable Equivalent Transformation for Quantization of LLMs
Wenhua Cheng, Yiyang Cai, Kaokao Lv +1
As large language models (LLMs) become more prevalent, there is a growing need for new and improved quantization methods that can meet the computationalast layer demands of these m…
Efficient Post-training Quantization with FP8 Formats
Haihao Shen, Naveen Mellempudi, Xin He +3
Recent advances in deep learning methods such as LLMs and Diffusion models have created a need for improved quantization methods that can meet the computational demands of these mo…
Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs
Wenhua Cheng, Weiwei Zhang, Haihao Shen +4
Large Language Models (LLMs) have demonstrated exceptional proficiency in language-related tasks, but their deployment poses significant challenges due to substantial memory and st…
An Efficient Sparse Inference Software Accelerator for Transformer-based Language Models on CPUs
Haihao Shen, Hengyu Meng, Bo Dong +9
In recent years, Transformer-based language models have become the standard approach for natural language processing tasks. However, stringent throughput and latency requirements i…
Fast DistilBERT on CPUs
Haihao Shen, Ofir Zafrir, Bo Dong +7
Transformer-based language models have become the standard approach to solving natural language processing tasks. However, industry adoption usually requires the maximum throughput…
QuaLA-MiniLM: a Quantized Length Adaptive MiniLM
Shira Guskin, Moshe Wasserblat, Chang Wang +1
Limited computational budgets often prevent transformers from being used in production and from having their high accuracy utilized. A knowledge distillation approach addresses the…
Prune Once for All: Sparse Pre-Trained Language Models
Ofir Zafrir, Ariel Larey, Guy Boudoukh +2
Transformer-based language models are applied to a wide range of applications in natural language processing. However, they are inefficient and difficult to deploy. In recent years…
Highly Efficient 8-bit Low Precision Inference of Convolutional Neural Networks with IntelCaffe
Jiong Gong, Haihao Shen, Guoming Zhang +6
High throughput and low latency inference of deep neural networks are critical for the deployment of deep learning applications. This paper presents the efficient inference techniq…