6 papers · 1 filter
Normalized Architectures are Natively 4-Bit
Maxim Fishman, Brian Chmiel, Ron Banner +2
Training large language models at 4-bit precision is critical for efficiency. We show that nGPT, an architecture that constrains weights and hidden representations to the unit hype…
FP4 All the Way: Fully Quantized Training of LLMs
Brian Chmiel, Maxim Fishman, Ron Banner +1
We demonstrate, for the first time, fully quantized training (FQT) of large language models (LLMs) using predominantly 4-bit floating-point (FP4) precision for weights, activations…
Scaling FP8 training to trillion-token LLMs
Maxim Fishman, Brian Chmiel, Ron Banner +1
We train, for the first time, large language models using FP8 precision on datasets up to 2 trillion tokens -- a 20-fold increase over previous limits. Through these extended train…
EXAQ: Exponent Aware Quantization For LLMs Acceleration
Moran Shkolnik, Maxim Fishman, Brian Chmiel +3
Quantization has established itself as the primary approach for decreasing the computational and storage expenses associated with Large Language Models (LLMs) inference. The majori…
Minimum Variance Unbiased N:M Sparsity for the Neural Gradients
Brian Chmiel, Itay Hubara, Ron Banner +1
In deep learning, fine-grained N:M sparsity reduces the data footprint and bandwidth of a General Matrix multiply (GEMM) up to x2, and doubles throughput by skipping computation of…
Accurate Neural Training with 4-bit Matrix Multiplications at Standard Formats
Brian Chmiel, Ron Banner, Elad Hoffer +2
Quantization of the weights and activations is one of the main methods to reduce the computational footprint of Deep Neural Networks (DNNs) training. Current methods enable 4-bit q…