LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale

Abstract

Large language models have been widely adopted but require significant GPU memory for inference. We develop a procedure for Int8 matrix multiplication for feed-forward and attention projection layers in transformers, which cut the memory needed for inference by half while retaining full precision performance. With our method, a 175B parameter 16/32-bit checkpoint can be loaded, converted to Int8, and used immediately without performance degradation. This is made possible by understanding and working around properties of highly systematic emergent features in transformer language models that dominate attention and transformer predictive performance. To cope with these features, we develop a two-part quantization procedure, LLM.int8(). We first use vector-wise quantization with separate normalization constants for each inner product in the matrix multiplication, to quantize most of the features. However, for the emergent outliers, we also include a new mixed-precision decomposition scheme, which isolates the outlier feature dimensions into a 16-bit matrix multiplication while still more than 99.9% of values are multiplied in 8-bit. Using LLM.int8(), we show empirically it is possible to perform inference in LLMs with up to 175B parameters without any performance degradation. This result makes such models much more accessible, for example making it possible to use OPT-175B/BLOOM on a single server with consumer GPUs. We open-source our software.

arXiv commentPublished at NeurIPS 2022. Camera-ready version

Cited by in corpus (12)

• Reproducible scaling laws for contrastive language-image learning
• Tiny Machine Learning: Progress and Futures
• OliVe: Accelerating Large Language Models via Hardware-friendly Outlier-Victim Pair Quantization
• Enhancing Phenotype Recognition in Clinical Notes Using Large Language Models: PhenoBCBERT and PhenoGPT
• LLM-FP4: 4-Bit Floating-Point Quantized Transformers
• Enhancing textual textbook question answering with large language models and retrieval augmented generation
• Allo: A Programming Model for Composable Accelerator Design
• Task-Oriented Semantic Communication in Large Multimodal Models-based Vehicle Networks
• Improved particle-flow event reconstruction with scalable neural networks for current and future particle detectors
• Revisiting Block-based Quantisation: What is Important for Sub-8-bit LLM Inference?
• WWW: What, When, Where to Compute-in-Memory
• ModuLoRA: Finetuning 2-Bit LLMs on Consumer GPUs by Integrating with Modular Quantizers