81 citations · 344 across the 26 of their papers we have counts for
9 papers · 1 filter
The LLM Surgeon
Tycho F. A. van der Ouderaa, Markus Nagel, Mart van Baalen +2
State-of-the-art language models are becoming increasingly large in an effort to achieve the highest performance on large corpora of available textual data. However, the sheer size…
MobileNVC: Real-time 1080p Neural Video Compression on a Mobile Device
Ties van Rozendaal, Tushar Singhal, Hoang Le +10
Neural video codecs have recently become competitive with standard codecs such as HEVC in the low-delay setting. However, most neural codecs are large floating-point networks that…
Softmax Bias Correction for Quantized Generative Models
Nilesh Prasad Pandey, Marios Fournarakis, Chirag Patel +1
Post-training quantization (PTQ) is the go-to compression technique for large generative models, such as stable diffusion or large language models. PTQ methods commonly keep the so…
ResQ: Residual Quantization for Video Perception
Davide Abati, Haitam Ben Yahia, Markus Nagel +1
This paper accelerates video perception, such as semantic segmentation and human pose estimation, by levering cross-frame redundancies. Unlike the existing approaches, which avoid…
QBitOpt: Fast and Accurate Bitwidth Reallocation during Training
Jorn Peters, Marios Fournarakis, Markus Nagel +2
Quantizing neural networks is one of the most effective methods for achieving efficient inference on mobile and embedded devices. In particular, mixed precision quantized (MPQ) net…
Pruning vs Quantization: Which is Better?
Andrey Kuzmin, Markus Nagel, Mart van Baalen +2
Neural network pruning and quantization techniques are almost as old as neural networks themselves. However, to date only ad-hoc comparisons between the two have been published. In…
Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothing
Yelysei Bondarenko, Markus Nagel, Tijmen Blankevoort
Transformer models have been widely adopted in various domains over the last years, and especially large language models have advanced the field of AI significantly. Due to their s…
FP8 versus INT8 for efficient deep learning inference
Mart van Baalen, Andrey Kuzmin, Suparna S Nair +8
Recently, the idea of using FP8 as a number format for neural network training has been floating around the deep learning world. Given that most training is currently conducted wit…
A Practical Mixed Precision Algorithm for Post-Training Quantization
Nilesh Prasad Pandey, Markus Nagel, Mart van Baalen +3
Neural network quantization is frequently used to optimize model size, latency and power consumption for on-device deployment of neural networks. In many cases, a target bit-width…