123 citations · 137 across the 13 of their papers we have counts for
6 papers · 1 filter
QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs
Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian L. Croci +6
We introduce QuaRot, a new Quantization scheme based on Rotations, which is able to quantize LLMs end-to-end, including all weights, activations, and KV cache in 4 bits. QuaRot rot…
Social Learning: Towards Collaborative Learning with Large Language Models
Amirkeivan Mohtashami, Florian Hartmann, Sian Gooding +3
We introduce the framework of "social learning" in the context of large language models (LLMs), whereby models share knowledge with each other in a privacy-aware manner using natur…
Special Properties of Gradient Descent with Large Learning Rates
Amirkeivan Mohtashami, Martin Jaggi, Sebastian Stich
When training neural networks, it has been widely observed that a large step size is essential in stochastic gradient descent (SGD) for obtaining superior models. However, the effe…
Characterizing & Finding Good Data Orderings for Fast Convergence of Sequential Gradient Methods
Amirkeivan Mohtashami, Sebastian Stich, Martin Jaggi
While SGD, which samples from the data with replacement is widely studied in theory, a variant called Random Reshuffling (RR) is more common in practice. RR iterates through random…
Masked Training of Neural Networks with Partial Gradients
Amirkeivan Mohtashami, Martin Jaggi, Sebastian U. Stich
State-of-the-art training algorithms for deep learning models are based on stochastic gradient descent (SGD). Recently, many variations have been explored: perturbing parameters fo…
Critical Parameters for Scalable Distributed Learning with Large Batches and Asynchronous Updates
Sebastian U. Stich, Amirkeivan Mohtashami, Martin Jaggi
It has been experimentally observed that the efficiency of distributed training with stochastic gradient (SGD) depends decisively on the batch size and -- in asynchronous implement…