Revisiting Block-based Quantisation: What is Important for Sub-8-bit LLM Inference?

Abstract

The inference of Large language models (LLMs) requires immense computation and memory resources. To curtail these costs, quantisation has merged as a promising solution, but existing LLM quantisation mainly focuses on 8-bit. In this work, we explore the statistical and learning properties of the LLM layer and attribute the bottleneck of LLM quantisation to numerical scaling offsets. To address this, we adapt block quantisations for LLMs, a family of methods that share scaling factors across packed numbers. Block quantisations efficiently reduce the numerical scaling offsets solely from an arithmetic perspective, without additional treatments in the computational path. Our nearly-lossless quantised 6-bit LLMs achieve a $19\times$ higher arithmetic density and $5\times$ memory density than the float32 baseline, surpassing the prior art 8-bit quantisation by $2.5\times$ in arithmetic density and $1.2\times$ in memory density, without requiring any data calibration or re-training. We also share our insights into sub-8-bit LLM quantisation, including the mismatch between activation and weight distributions, optimal fine-tuning strategies, and a lower quantisation granularity inherent in the statistical properties of LLMs. The latter two tricks enable nearly-lossless 4-bit LLMs on downstream tasks. Our code is open-sourced.

arXiv commentAccepted by EMNLP2023

References in corpus (16)

• LLaMA: Open and Efficient Foundation Language Models
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• Quantizing deep convolutional networks for efficient inference: A whitepaper
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
• Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
• Towards Accurate Binary Convolutional Neural Network
• Convolutional Neural Networks using Logarithmic Data Representation
• Training and Inference with Integers in Deep Neural Networks
• Mixed Precision Quantization of ConvNets via Differentiable Neural Architecture Search
• GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
• LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale
• SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
• ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers
• BinaryBERT: Pushing the Limit of BERT Quantization
• VS-Quant: Per-vector Scaled Quantization for Accurate Low-Precision Neural Network Inference
• A Dataflow Compiler for Efficient LLM Inference using Custom Microscaling Formats