732 citations · 1.7k across the 44 of their papers we have counts for
7 papers · 1 filter
Why are Adaptive Methods Good for Attention Models?
Jingzhao Zhang, Sai Praneeth Karimireddy, Andreas Veit +4
While stochastic gradient descent (SGD) is still the \emph{de facto} algorithm in deep learning, adaptive methods like Clipped SGD/Adam have been observed to outperform SGD across…
SCAFFOLD: Stochastic Controlled Averaging for Federated Learning
Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri +3
Federated Averaging (FedAvg) has emerged as the algorithm of choice for federated learning due to its simplicity and low communication cost. However, in spite of recent research ef…
The Error-Feedback Framework: Better Rates for SGD with Delayed Gradients and Compressed Communication
Sebastian U. Stich, Sai Praneeth Karimireddy
We analyze (stochastic) gradient descent (SGD) with delayed updates on smooth quasi-convex and non-convex functions and derive concise, non-asymptotic, convergence rates. We show t…
Amplifying Rényi Differential Privacy via Shuffling
Eloïse Berthier, Sai Praneeth Karimireddy
Differential privacy is a useful tool to build machine learning models which do not release too much information about the training data. We study the Rényi differential privacy of…
PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimization
Thijs Vogels, Sai Praneeth Karimireddy, Martin Jaggi
We study gradient compression methods to alleviate the communication bottleneck in data-parallel distributed optimization. Despite the significant attention received, current compr…
Accelerating Gradient Boosting Machine
Haihao Lu, Sai Praneeth Karimireddy, Natalia Ponomareva +1
Gradient Boosting Machine (GBM) is an extremely powerful supervised learning algorithm that is widely used in practice. GBM routinely features as a leading algorithm in machine lea…
Error Feedback Fixes SignSGD and other Gradient Compression Schemes
Sai Praneeth Karimireddy, Quentin Rebjock, Sebastian U. Stich +1
Sign-based algorithms (e.g. signSGD) have been proposed as a biased gradient compression technique to alleviate the communication bottleneck in training large neural networks acros…