6 papers
Activation Denoising: A Robustness View on Parallel vs Sequential LLM Quantization
Yan Scholten, Rachel Lawrence, James Hensman +3
Post-training quantization is a powerful tool for compressing large language models. The most scalable methods quantize every layer in parallel, but quantization errors then compou…
Rotated Manifold Optimization for Low-Rank Adaptation
Yuhui Ding, Javier Zazo, James Hensman
We propose a novel optimizer for low-rank adaptation (LoRA) that explicitly incorporates the gauge symmetry of low-rank factorization. Our optimizer extends recent matrix optimizer…
Revisiting Transformer Layer Parameterization Through Causal Energy Minimization
Jin Xu, Camille Couturier, Victor Rühle +2
Transformer blocks typically combine multi-head attention (MHA) for token mixing with gated MLPs for token-wise feature transformation, yet many choices in their parameterization r…
OptRot: Mitigating Weight Outliers via Data-Free Rotations for Post-Training Quantization
Advait Gadhikar, Riccardo Grazzi, James Hensman
The presence of outliers in Large Language Models (LLMs) weights and activations makes them difficult to quantize. Recent work has leveraged rotations to mitigate these outliers. I…
TurboAttention: Efficient Attention Approximation For High Throughputs LLMs
Hao Kang, Srikant Bharadwaj, James Hensman +3
Large language model (LLM) inference demands significant amount of computation and memory, especially in the key attention mechanism. While techniques, such as quantization and acc…
Low-Rank Correction for Quantized LLMs
Meyer Scetbon, James Hensman
We consider the problem of model compression for Large Language Models (LLMs) at post-training time, where the task is to compress a well-trained model using only a small set of ca…