704 citations · 4.8k across the 117 of their papers we have counts for
15 papers · 1 filter
Learning from History for Byzantine Robust Optimization
Sai Praneeth Karimireddy, Lie He, Martin Jaggi
Byzantine robustness has received significant attention recently given its importance for distributed and federated learning. In spite of this, we identify severe flaws in existing…
A Linearly Convergent Algorithm for Decentralized Optimization: Sending Less Bits for Free!
Dmitry Kovalev, Anastasia Koloskova, Martin Jaggi +2
Decentralized optimization methods enable on-device training of machine learning models without a central coordinator. In many scenarios communication between devices is energy dem…
Sparse Communication for Training Deep Networks
Negar Foroutan Eghlidi, Martin Jaggi
Synchronous stochastic gradient descent (SGD) is the most common method used for distributed training of deep learning models. In this algorithm, each worker shares its local gradi…
Mime: Mimicking Centralized Stochastic Algorithms in Federated Learning
Sai Praneeth Karimireddy, Martin Jaggi, Satyen Kale +4
Federated learning (FL) is a challenging setting for optimization due to the heterogeneity of the data across different clients which gives rise to the client drift phenomenon. In…
PowerGossip: Practical Low-Rank Communication Compression in Decentralized Deep Learning
Thijs Vogels, Sai Praneeth Karimireddy, Martin Jaggi
Lossy gradient compression has become a practical tool to overcome the communication bottleneck in centrally coordinated distributed training of machine learning models. However, a…
Dynamic Model Pruning with Feedback
Tao Lin, Sebastian U. Stich, Luis Barba +2
Deep neural networks often have millions of parameters. This can hinder their deployment to low-end devices, not only due to high memory requirements but also because of increased…
Extrapolation for Large-batch Training in Deep Learning
Tao Lin, Lingjing Kong, Sebastian U. Stich +1
Deep learning networks are typically trained by Stochastic Gradient Descent (SGD) methods that iteratively improve the model parameters by estimating a gradient on a very small fra…
Multi-Head Attention: Collaborate Instead of Concatenate
Jean-Baptiste Cordonnier, Andreas Loukas, Martin Jaggi
Attention layers are widely used in natural language processing (NLP) and are beginning to influence computer vision architectures. Training very large transformer models allowed s…
Taming GANs with Lookahead-Minmax
Tatjana Chavdarova, Matteo Pagliardini, Sebastian U. Stich +2
Generative Adversarial Networks are notoriously challenging to train. The underlying minmax optimization is highly susceptible to the variance of the stochastic gradient and the ro…
Byzantine-Robust Learning on Heterogeneous Datasets via Bucketing
Sai Praneeth Karimireddy, Lie He, Martin Jaggi
In Byzantine robust distributed or federated learning, a central server wants to train a machine learning model over data distributed across multiple workers. However, a fraction o…
Ensemble Distillation for Robust Model Fusion in Federated Learning
Tao Lin, Lingjing Kong, Sebastian U. Stich +1
Federated Learning (FL) is a machine learning setting where many devices collaboratively train a machine learning model while keeping the training data decentralized. In most of th…
Secure Byzantine-Robust Machine Learning
Lie He, Sai Praneeth Karimireddy, Martin Jaggi
Increasingly machine learning systems are being deployed to edge servers and devices (e.g. mobile phones) and trained in a collaborative manner. Such distributed/federated/decentra…
Masking as an Efficient Alternative to Finetuning for Pretrained Language Models
Mengjie Zhao, Tao Lin, Fei Mi +2
We present an efficient method of utilizing pretrained language models, where we learn selective binary masks for pretrained weights in lieu of modifying them through finetuning. E…
Understanding the Effects of Data Parallelism and Sparsity on Neural Network Training
Namhoon Lee, Thalaiyasingam Ajanthan, Philip H. S. Torr +1
We study two factors in neural network training: data parallelism and sparsity; here, data parallelism means processing training data in parallel using distributed systems (or equi…
A Unified Theory of Decentralized SGD with Changing Topology and Local Updates
Anastasia Koloskova, Nicolas Loizou, Sadra Boreiri +2
Decentralized stochastic optimization methods have gained a lot of attention recently, mainly because of their cheap per iteration cost, data locality, and their communication-effi…