COLA: Decentralized Linear Learning

Abstract

Decentralized machine learning is a promising emerging paradigm in view of global challenges of data ownership and privacy. We consider learning of linear classification and regression models, in the setting where the training data is decentralized over many user devices, and the learning algorithm must run on-device, on an arbitrary communication network, without a central coordinator. We propose COLA, a new decentralized training algorithm with strong theoretical guarantees and superior practical performance. Our framework overcomes many limitations of existing methods, and achieves communication efficiency, scalability, elasticity as well as resilience to changes in data and participating devices.

Cited by in corpus (17)

• Robust Aggregation for Federated Learning
• Communication-Efficient Distributed Deep Learning: A Comprehensive Survey
• A Linearly Convergent Proximal Gradient Algorithm for Decentralized Optimization
• Federated Learning With Quantized Global Model Updates
• Turbo-Aggregate: Breaking the Quadratic Aggregation Barrier in Secure Federated Learning
• Privacy Preservation in Federated Learning: An insightful survey from the GDPR Perspective
• Decentralized Stochastic Gradient Langevin Dynamics and Hamiltonian Monte Carlo
• Network-Aware Optimization of Distributed Learning for Fog Computing
• On the Benefits of Multiple Gossip Steps in Communication-Constrained Decentralized Optimization
• Byzantine-resilient Decentralized Stochastic Gradient Descent
• A Linearly Convergent Algorithm for Decentralized Optimization: Sending Less Bits for Free!
• Federated CycleGAN for Privacy-Preserving Image-to-Image Translation
• Decentralized Learning with Lazy and Approximate Dual Gradients
• Decentralized Statistical Inference with Unrolled Graph Neural Networks
• Decentralized Feature-Distributed Optimization for Generalized Linear Models
• Distributed Networked Learning with Correlated Data
• Generalized ADMM in Distributed Learning via Variational Inequality