Privacy-preserving Incremental ADMM for Decentralized Consensus Optimization
arXiv:2003.10615 · doi:10.1109/TSP.2020.3027917
Abstract
The alternating direction method of multipliers (ADMM) has been recently recognized as a promising optimizer for large-scale machine learning models. However, there are very few results studying ADMM from the aspect of communication costs, especially jointly with privacy preservation, which are critical for distributed learning. We investigate the communication efficiency and privacy-preservation of ADMM by solving the consensus optimization problem over decentralized networks. Since walk algorithms can reduce communication load, we first propose incremental ADMM (I-ADMM) based on the walk algorithm, the updating order of which follows a Hamiltonian cycle instead. However, I-ADMM cannot guarantee the privacy for agents against external eavesdroppers even if the randomized initialization is applied. To protect privacy for agents, we then propose two privacy-preserving incremental ADMM algorithms, i.e., PI-ADMM1 and PI-ADMM2, where perturbation over step sizes and primal variables is adopted, respectively. Through theoretical analyses, we prove the convergence and privacy preservation for PI-ADMM1, which are further supported by numerical experiments. Besides, simulations demonstrate that the proposed PI-ADMM1 and PI-ADMM2 algorithms are communication efficient compared with state-of-the-art methods.
References in corpus (16)
- Diffusion Adaptation Strategies for Distributed Optimization and Learning over Networks
- Sparse Communication for Distributed Gradient Descent
- Multi-Agent Distributed Optimization via Inexact Consensus ADMM
- D-ADMM: A Communication-Efficient Distributed Algorithm For Separable Optimization
- Differentially Private Distributed Constrained Optimization
- Asynchronous Distributed ADMM for Large-Scale Optimization- Part I: Algorithm and Convergence Analysis
- Monitoring and Optimization for Power Grids: A Signal Processing Perspective
- DP-ADMM: ADMM-based Distributed Learning with Differential Privacy
- Error Feedback Fixes SignSGD and other Gradient Compression Schemes
- Distributed Discrete-time Optimization in Multi-agent Networks Using only Sign of Relative State
- Improving the Privacy and Accuracy of ADMM-Based Distributed Algorithms
- Quantized Consensus ADMM for Multi-Agent Distributed Optimization
- Private Learning on Networks: Part II
- Walkman: A Communication-Efficient Random-Walk Algorithm for Decentralized Optimization
- Decentralized Multi-Task Learning Based on Extreme Learning Machines
- Local Differential Privacy in Decentralized Optimization