On Consensus-Optimality Trade-offs in Collaborative Deep Learning

Abstract

In distributed machine learning, where agents collaboratively learn from diverse private data sets, there is a fundamental tension between consensus and optimality. In this paper, we build on recent algorithmic progresses in distributed deep learning to explore various consensus-optimality trade-offs over a fixed communication topology. First, we propose the incremental consensus-based distributed SGD (i-CDSGD) algorithm, which involves multiple consensus steps (where each agent communicates information with its neighbors) within each SGD iteration. Second, we propose the generalized consensus-based distributed SGD (g-CDSGD) algorithm that enables us to navigate the full spectrum from complete consensus (all agents agree) to complete disagreement (each agent converges to individual model parameters). We analytically establish convergence of the proposed algorithms for strongly convex and nonconvex objective functions; we also analyze the momentum variants of the algorithms for the strongly convex case. We support our algorithms via numerical experiments, and demonstrate significant improvements over existing methods for collaborative deep learning.

References in corpus (8)

• Communication-Efficient Learning of Deep Networks from Decentralized Data
• TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning
• Gradient Sparsification for Communication-Efficient Distributed Optimization
• Optimal algorithms for smooth and strongly convex distributed optimization in networks
• Distributed Deep Learning Using Synchronous Stochastic Gradient Descent
• Gossip training for deep learning
• Adaptive Consensus ADMM for Distributed Optimization
• Balancing Communication and Computation in Distributed Optimization

Cited by in corpus (2)

• Consensus Control for Decentralized Deep Learning
• Cross-Gradient Aggregation for Decentralized Learning from Non-IID data