6 papers
TWLA: Achieving Ternary Weights and Low-Bit Activations for LLMs via Post-Training Quantization
Zhixiong Zhao, Zukang Xu, Zhixuan Chen +3
Large language models (LLMs) exhibit exceptional general language processing capabilities, but their memory and compute costs hinder deployment. Ternarization has emerged as a prom…
NLI:Non-uniform Linear Interpolation Approximation of Nonlinear Operations for Efficient LLMs Inference
Jiangyong Yu, Xiaomeng Han, Xing Hu +3
Large Language Models (LLMs) have demonstrated remarkable performance across a wide range of tasks, but their deployment is often constrained by substantial memory footprints and c…
BBAL: A Bidirectional Block Floating Point-Based Quantisation Accelerator for Large Language Models
Xiaomeng Han, Yuan Cheng, Jing Wang +6
Large language models (LLMs), with their billions of parameters, pose substantial challenges for deployment on edge devices, straining both memory capacity and computational resour…
NVR: Vector Runahead on NPUs for Sparse Memory Access
Hui Wang, Zhengpeng Zhao, Jing Wang +11
Deep Neural Networks are increasingly leveraging sparsity to reduce the scaling up of model parameter size. However, reducing wall-clock time through sparsity and pruning remains c…
Pushing the Limits of BFP on Narrow Precision LLM Inference
Hui Wang, Yuan Cheng, Xiaomeng Han +3
The substantial computational and memory demands of Large Language Models (LLMs) hinder their deployment. Block Floating Point (BFP) has proven effective in accelerating linear ope…
OstQuant: Refining Large Language Model Quantization with Orthogonal and Scaling Transformations for Better Distribution Fitting
Xing Hu, Yuan Cheng, Dawei Yang +6
Post-training quantization (PTQ) has emerged as a widely adopted technique for compressing and accelerating Large Language Models (LLMs). The major challenge in LLM quantization is…