2 citations · 3 across the 17 of their papers we have counts for
14 papers · 1 filter
SQuAT: Spherical Lattice Vector Quantization-Aware Training
Sebastian Loeschcke, Ivan Koryakovskiy, Tycho F. A. van der Ouderaa +4
Vector quantization (VQ), in particular lattice-based methods, offer strong rate-distortion trade-offs for low-bit large language models (LLMs), yet their behavior under quantizati…
The Key to Going Linear: Analysis-Driven Transformer Linearization
Anna Kuzina, Paul N. Whatmough, Babak Ehteshami Bejnordi
The quadratic cost of causal self-attention severely bottlenecks long-context transformer inference. While numerous post hoc linearization pipelines exist, it is difficult to ident…
Dissecting Quantization Error: A Concentration-Alignment Perspective
Marco Federici, Boris van Breugel, Paul Whatmough +1
Quantization can drastically increase the efficiency of large language and vision models, but typically incurs an accuracy drop. Recently, function-preserving transforms (e.g. rota…
Efficient Reasoning on the Edge
Yelysei Bondarenko, Thomas Hehn, Rob Hesselink +15
Large language models (LLMs) with chain-of-thought reasoning achieve state-of-the-art performance across complex problem-solving tasks, but their verbose reasoning traces and large…
Leech Lattice Vector Quantization for Efficient LLM Compression
Tycho F. A. van der Ouderaa, Mart van Baalen, Paul Whatmough +1
Scalar quantization of large language models (LLMs) is fundamentally limited by information-theoretic bounds. While vector quantization (VQ) overcomes these limits by encoding bloc…
STaMP: Sequence Transformation and Mixed Precision for Low-Precision Activation Quantization
Marco Federici, Riccardo Del Chiaro, Boris van Breugel +2
Quantization is the key method for reducing inference latency, power and memory footprint of generative AI models. However, accuracy often degrades sharply when activations are qua…
KaVa: Latent Reasoning via Compressed KV-Cache Distillation
Anna Kuzina, Maciej Pioro, Paul N. Whatmough +1
Large Language Models (LLMs) excel at multi-step reasoning problems with explicit chain-of-thought (CoT), but verbose traces incur significant computational costs and memory overhe…
Dirichlet-Prior Shaping: Guiding Expert Specialization in Upcycled MoEs
Leyla Mirvakhabova, Babak Ehteshami Bejnordi, Gaurav Kumar +3
Upcycling pre-trained dense models into sparse Mixture-of-Experts (MoEs) efficiently increases model capacity but often suffers from poor expert specialization due to naive weight…
FPTQuant: Function-Preserving Transforms for LLM Quantization
Boris van Breugel, Yelysei Bondarenko, Paul Whatmough +1
Large language models (LLMs) require substantial compute, and thus energy, at inference time. While quantizing weights and activations is effective at improving efficiency, naive q…
Efficient LLM Inference using Dynamic Input Pruning and Cache-Aware Masking
Marco Federici, Davide Belli, Mart van Baalen +5
While mobile devices provide ever more compute power, improvements in DRAM bandwidth are much slower. This is unfortunate for large language model (LLM) token generation, which is…
Mixture of Cache-Conditional Experts for Efficient Mobile Device Inference
Andrii Skliar, Ties van Rozendaal, Romain Lepert +5
Mixture of Experts (MoE) LLMs have recently gained attention for their ability to enhance performance by selectively engaging specialized subnetworks or "experts" for each input. H…
Rapid Switching and Multi-Adapter Fusion via Sparse High Rank Adapters
Kartikeya Bhardwaj, Nilesh Prasad Pandey, Sweta Priyadarshi +9
In this paper, we propose Sparse High Rank Adapters (SHiRA) that directly finetune 1-2% of the base model weights while leaving others unchanged, thus, resulting in a highly sparse…
Sparse High Rank Adapters
Kartikeya Bhardwaj, Nilesh Prasad Pandey, Sweta Priyadarshi +9
Low Rank Adaptation (LoRA) has gained massive attention in the recent generative AI research. One of the main advantages of LoRA is its ability to be fused with pretrained models,…
GPTVQ: The Blessing of Dimensionality for LLM Quantization
Mart van Baalen, Andrey Kuzmin, Ivan Koryakovskiy +6
In this work we show that the size versus accuracy trade-off of neural network quantization can be significantly improved by increasing the quantization dimensionality. We propose…