13 papers
Clipping Makes Distributed and Federated Asynchronous SGD Robust to Stragglers
Samuel Erickson, Mikael Johansson
In modern machine learning, parallelization of training is an important strategy for increasing scale. Asynchronous stochastic gradient descent (ASGD), which maximizes the utilizat…
Decentralized SGD with Controlled Disagreement Finds Flatter Minima
Zesen Wang, Mikael Johansson
Decentralized training is often regarded as inferior to centralized training because the consensus errors between workers are thought to undermine convergence and generalization. T…
Byzantine-Robust Federated Learning with Learnable Aggregation Weights
Javad Parsa, Amir Hossein Daghestani, André M. H. Teixeira +1
Federated Learning (FL) enables clients to collaboratively train a global model without sharing their private data. However, the presence of malicious (Byzantine) clients poses sig…
Personalized Federated Learning under Model Dissimilarity Constraints
Samuel Erickson, Mikael Johansson
One of the defining challenges in federated learning is that of statistical heterogeneity among clients. We address this problem with KARULA, a regularized strategy for personalize…
Non-convex composite federated learning with heterogeneous data
Jiaojiao Zhang, Jiang Hu, Mikael Johansson
We propose an innovative algorithm for non-convex composite federated learning that decouples the proximal operator evaluation and the communication between server and clients. Mor…
Locally Differentially Private Online Federated Learning With Correlated Noise
Jiaojiao Zhang, Linglingzhi Zhu, Dominik Fay +1
We introduce a locally differentially private (LDP) algorithm for online federated learning that employs temporally correlated noise to improve utility while preserving privacy. To…
From promise to practice: realizing high-performance decentralized training
Zesen Wang, Jiaojiao Zhang, Xuyang Wu +1
Decentralized training of deep neural networks has attracted significant attention for its theoretically superior scalability over synchronous data-parallel methods like All-Reduce…
Temporal Predictive Coding for Gradient Compression in Distributed Learning
Adrian Edin, Zheng Chen, Michel Kieffer +1
This paper proposes a prediction-based gradient compression method for distributed learning with event-triggered communication. Our goal is to reduce the amount of information tran…
Nonconvex Federated Learning on Compact Smooth Submanifolds With Heterogeneous Data
Jiaojiao Zhang, Jiang Hu, Anthony Man-Cho So +1
Many machine learning tasks, such as principal component analysis and low-rank matrix completion, give rise to manifold optimization problems. Although there is a large body of wor…
Novel Limited Memory Quasi-Newton Methods Based On Optimal Matrix Approximation
Erik Berglund, Mikael Johansson
Update formulas for the Hessian approximations in quasi-Newton methods such as BFGS can be derived as analytical solutions to certain nearest-matrix problems. In this article, we p…
Soft quasi-Newton: Guaranteed positive definiteness by relaxing the secant constraint
Erik Berglund, Jiaojiao Zhang, Mikael Johansson
We propose a novel algorithm, termed soft quasi-Newton (soft QN), for optimization in the presence of bounded noise. Traditional quasi-Newton algorithms are vulnerable to such pert…
Differentially Private Online Federated Learning with Correlated Noise
Jiaojiao Zhang, Linglingzhi Zhu, Mikael Johansson
We introduce a novel differentially private algorithm for online federated learning that employs temporally correlated noise to enhance utility while ensuring privacy of continuous…
Composite federated learning with heterogeneous data
Jiaojiao Zhang, Jiang Hu, Mikael Johansson
We propose a novel algorithm for solving the composite Federated Learning (FL) problem. This algorithm manages non-smooth regularization by strategically decoupling the proximal op…