81 citations · 344 across the 26 of their papers we have counts for
25 papers · 1 filter
Myosotis: structured computation for attention like layer
Evgenii Egorov, Hanno Ackermann, Markus Nagel +1
Attention layers apply a sequence-to-sequence mapping whose parameters depend on the pairwise interactions of the input elements. However, without any structural assumptions, memor…
How to Parameterize Asymmetric Quantization Ranges for Quantization-Aware Training
Jaeseong You, Minseop Park, Kyunggeun Lee +3
This paper investigates three different parameterizations of asymmetric uniform quantization for quantization-aware training: (1) scale and offset, (2) minimum and maximum, and (3)…
GPTVQ: The Blessing of Dimensionality for LLM Quantization
Mart van Baalen, Andrey Kuzmin, Ivan Koryakovskiy +6
In this work we show that the size versus accuracy trade-off of neural network quantization can be significantly improved by increasing the quantization dimensionality. We propose…
The LLM Surgeon
Tycho F. A. van der Ouderaa, Markus Nagel, Mart van Baalen +2
State-of-the-art language models are becoming increasingly large in an effort to achieve the highest performance on large corpora of available textual data. However, the sheer size…
Softmax Bias Correction for Quantized Generative Models
Nilesh Prasad Pandey, Marios Fournarakis, Chirag Patel +1
Post-training quantization (PTQ) is the go-to compression technique for large generative models, such as stable diffusion or large language models. PTQ methods commonly keep the so…
QBitOpt: Fast and Accurate Bitwidth Reallocation during Training
Jorn Peters, Marios Fournarakis, Markus Nagel +2
Quantizing neural networks is one of the most effective methods for achieving efficient inference on mobile and embedded devices. In particular, mixed precision quantized (MPQ) net…
Pruning vs Quantization: Which is Better?
Andrey Kuzmin, Markus Nagel, Mart van Baalen +2
Neural network pruning and quantization techniques are almost as old as neural networks themselves. However, to date only ad-hoc comparisons between the two have been published. In…
Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothing
Yelysei Bondarenko, Markus Nagel, Tijmen Blankevoort
Transformer models have been widely adopted in various domains over the last years, and especially large language models have advanced the field of AI significantly. Due to their s…
FP8 versus INT8 for efficient deep learning inference
Mart van Baalen, Andrey Kuzmin, Suparna S Nair +8
Recently, the idea of using FP8 as a number format for neural network training has been floating around the deep learning world. Given that most training is currently conducted wit…
A Practical Mixed Precision Algorithm for Post-Training Quantization
Nilesh Prasad Pandey, Markus Nagel, Mart van Baalen +3
Neural network quantization is frequently used to optimize model size, latency and power consumption for on-device deployment of neural networks. In many cases, a target bit-width…
Quadapter: Adapter for GPT-2 Quantization
Minseop Park, Jaeseong You, Markus Nagel +1
Transformer language models such as GPT-2 are difficult to quantize because of outliers in activations leading to a large quantization error. To adapt to the error, one must use qu…
FP8 Quantization: The Power of the Exponent
Andrey Kuzmin, Mart Van Baalen, Yuwei Ren +3
When quantizing neural networks for efficient inference, low-bit integers are the go-to format for efficiency. However, low-bit floating point numbers have an extra degree of freed…
Quantized Sparse Weight Decomposition for Neural Network Compression
Andrey Kuzmin, Mart van Baalen, Markus Nagel +1
In this paper, we introduce a novel method of neural network weight compression. In our method, we store weight tensors as sparse, quantized matrix factors, whose product is comput…
Quantization Robust Federated Learning for Efficient Inference on Heterogeneous Devices
Kartik Gupta, Marios Fournarakis, Matthias Reisser +2
Federated Learning (FL) is a machine learning paradigm to distributively learn machine learning models from decentralized data that remains on-device. Despite the success of standa…
Overcoming Oscillations in Quantization-Aware Training
Markus Nagel, Marios Fournarakis, Yelysei Bondarenko +1
When training neural networks with simulated quantization, we observe that quantized weights can, rather unexpectedly, oscillate between two grid-points. The importance of this eff…
Cyclical Pruning for Sparse Neural Networks
Suraj Srinivas, Andrey Kuzmin, Markus Nagel +3
Current methods for pruning neural network weights iteratively apply magnitude-based pruning on the model weights and re-train the resulting model to recover lost accuracy. In this…
Neural Network Quantization with AI Model Efficiency Toolkit (AIMET)
Sangeetha Siddegowda, Marios Fournarakis, Markus Nagel +3
While neural networks have advanced the frontiers in many machine learning applications, they often come at a high computational cost. Reducing the power and latency of neural netw…
Implicit Neural Video Compression
Yunfan Zhang, Ties van Rozendaal, Johann Brehmer +2
We propose a method to compress full-resolution video sequences with implicit neural representations. Each frame is represented as a neural network that maps coordinate positions t…
Understanding and Overcoming the Challenges of Efficient Transformer Quantization
Yelysei Bondarenko, Markus Nagel, Tijmen Blankevoort
Transformer-based architectures have become the de-facto standard models for a wide range of Natural Language Processing tasks. However, their memory footprint and high latency are…
A White Paper on Neural Network Quantization
Markus Nagel, Marios Fournarakis, Rana Ali Amjad +3
While neural networks have advanced the frontiers in many applications, they often come at a high computational cost. Reducing the power and latency of neural network inference is…
In-Hindsight Quantization Range Estimation for Quantized Training
Marios Fournarakis, Markus Nagel
Quantization techniques applied to the inference of deep neural networks have enabled fast and efficient execution on resource-constraint devices. The success of quantization durin…
Bayesian Bits: Unifying Quantization and Pruning
Mart van Baalen, Christos Louizos, Markus Nagel +4
We introduce Bayesian Bits, a practical method for joint mixed precision quantization and pruning through gradient based optimization. Bayesian Bits employs a novel decomposition o…
Up or Down? Adaptive Rounding for Post-Training Quantization
Markus Nagel, Rana Ali Amjad, Mart van Baalen +2
When quantizing neural networks, assigning each floating-point weight to its nearest fixed-point value is the predominant approach. We find that, perhaps surprisingly, this is not…
Taxonomy and Evaluation of Structured Compression of Convolutional Neural Networks
Andrey Kuzmin, Markus Nagel, Saurabh Pitre +3
The success of deep neural networks in many real-world applications is leading to new challenges in building more efficient architectures. One effective way of making networks more…