Improving the Privacy and Accuracy of ADMM-Based Distributed Algorithms

Abstract

Alternating direction method of multiplier (ADMM) is a popular method used to design distributed versions of a machine learning algorithm, whereby local computations are performed on local data with the output exchanged among neighbors in an iterative fashion. During this iterative process the leakage of data privacy arises. A differentially private ADMM was proposed in prior work (Zhang & Zhu, 2017) where only the privacy loss of a single node during one iteration was bounded, a method that makes it difficult to balance the tradeoff between the utility attained through distributed computation and privacy guarantees when considering the total privacy loss of all nodes over the entire iterative process. We propose a perturbation method for ADMM where the perturbed term is correlated with the penalty parameters; this is shown to improve the utility and privacy simultaneously. The method is based on a modified ADMM where each node independently determines its own penalty parameter in every iteration and decouples it from the dual updating step size. The condition for convergence of the modified ADMM and the lower bound on the convergence rate are also derived.

arXiv commentaccepted to 35th International Conference on Machine Learning

References in corpus (6)

• On the Linear Convergence of the ADMM in Decentralized Consensus Optimization
• Differentially Private Distributed Constrained Optimization
• Differentially Private Empirical Risk Minimization: Efficient Algorithms and Tight Error Bounds
• Improving the Privacy and Accuracy of ADMM-Based Distributed Algorithms
• Adaptive ADMM with Spectral Penalty Parameter Selection
• ADMM Based Privacy-preserving Decentralized Optimization

Cited by in corpus (14)

• Privacy-Preserving Distributed Optimization via Subspace Perturbation: A General Framework
• Privacy-preserving Distributed Machine Learning via Local Randomization and ADMM Perturbation
• Improving the Privacy and Accuracy of ADMM-Based Distributed Algorithms
• Privacy-preserving Incremental ADMM for Decentralized Consensus Optimization
• Communication-Efficient ADMM-based Federated Learning
• Privacy-preserving Decentralized Aggregation for Federated Learning
• Differentially Private ADMM for Convex Distributed Learning: Improved Accuracy via Multi-Step Approximation
• Differentially Private ADMM for Distributed Medical Machine Learning
• Privacy-preserving Decentralized Optimization via Decomposition
• Recycled ADMM: Improve Privacy and Accuracy with Less Computation in Distributed Algorithms
• Differential Privacy for Sparse Classification Learning
• Muffliato: Peer-to-Peer Privacy Amplification for Decentralized Optimization and Averaging
• From Noisy Fixed-Point Iterations to Private ADMM for Centralized and Federated Learning
• Topology-aware Differential Privacy for Decentralized Image Classification