1.6k citations · 3.9k across the 52 of their papers we have counts for
6 papers · 1 filter
Differentially Private Adaptive Optimization with Delayed Preconditioners
Tian Li, Manzil Zaheer, Ken Ziyu Liu +3
Privacy noise may negate the benefits of using adaptive optimizers in differentially private model training. Prior works typically address this issue by using auxiliary information…
On the Algorithmic Stability and Generalization of Adaptive Optimization Methods
Han Nguyen, Hai Pham, Sashank J. Reddi +1
Despite their popularity in deep learning and machine learning in general, the theoretical properties of adaptive optimizers such as Adagrad, RMSProp, Adam or AdamW are not yet ful…
The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformers
Zonglin Li, Chong You, Srinadh Bhojanapalli +8
This paper studies the curious phenomenon for machine learning models with Transformer architectures that their activation maps are sparse. By activation map we refer to the interm…
FedLite: A Scalable Approach for Federated Learning on Resource-constrained Clients
Jianyu Wang, Hang Qi, Ankit Singh Rawat +4
In classical federated learning, the clients contribute to the overall training by communicating local updates for the underlying model on their private data to a coordinating serv…
Private Adaptive Optimization with Side Information
Tian Li, Manzil Zaheer, Sashank J. Reddi +1
Adaptive optimization methods have become the default solvers for many machine learning tasks. Unfortunately, the benefits of adaptivity may degrade when training with differential…
Robust Training of Neural Networks Using Scale Invariant Architectures
Zhiyuan Li, Srinadh Bhojanapalli, Manzil Zaheer +2
In contrast to SGD, adaptive gradient methods like Adam allow robust training of modern deep networks, especially large language models. However, the use of adaptivity not only com…