704 citations · 4.8k across the 117 of their papers we have counts for
3 papers · 1 filter
QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs
Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian L. Croci +6
We introduce QuaRot, a new Quantization scheme based on Rotations, which is able to quantize LLMs end-to-end, including all weights, activations, and KV cache in 4 bits. QuaRot rot…
DenseFormer: Enhancing Information Flow in Transformers via Depth Weighted Averaging
Matteo Pagliardini, Amirkeivan Mohtashami, Francois Fleuret +1
The transformer architecture by Vaswani et al. (2017) is now ubiquitous across application domains, from natural language processing to speech processing and image understanding. W…
Spectral Preconditioning for Gradient Methods on Graded Non-convex Functions
Nikita Doikov, Sebastian U. Stich, Martin Jaggi
The performance of optimization methods is often tied to the spectrum of the objective Hessian. Yet, conventional assumptions, such as smoothness, do often not enable us to make fi…