From Noisy Fixed-Point Iterations to Private ADMM for Centralized and Federated Learning

Abstract

We study differentially private (DP) machine learning algorithms as instances of noisy fixed-point iterations, in order to derive privacy and utility results from this well-studied framework. We show that this new perspective recovers popular private gradient-based methods like DP-SGD and provides a principled way to design and analyze new private optimization algorithms in a flexible manner. Focusing on the widely-used Alternating Directions Method of Multipliers (ADMM) method, we use our general framework to derive novel private ADMM algorithms for centralized, federated and fully decentralized learning. For these three algorithms, we establish strong privacy guarantees leveraging privacy amplification by iteration and by subsampling. Finally, we provide utility guarantees using a unified analysis that exploits a recent linear convergence result for noisy fixed-point iterations.

arXiv commentAccepted to ICML 2023. v3: added references to the first papers that introduced block-wise fixed-point iterations (Iutzeler et al., 2013; Bianchi et al., 2016)

References in corpus (16)

• Deep Learning with Differential Privacy
• Renyi Differential Privacy
• On the Linear Convergence of the ADMM in Decentralized Consensus Optimization
• Learning Differentially Private Recurrent Language Models
• Differentially Private Empirical Risk Minimization
• Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descent
• Privacy Amplification by Subsampling: Tight Analyses via Couplings and Divergences
• Fixed Point Strategies in Data Science
• Decentralized Collaborative Learning of Personalized Models over Networks
• Improving the Privacy and Accuracy of ADMM-Based Distributed Algorithms
• Learning Privately over Distributed Features: An ADMM Sharing Approach
• Differentially Private Federated Learning on Heterogeneous Data
• Bypassing the Ambient Dimension: Private SGD with Gradient Subspace Identification
• Privacy Amplification by Decentralization
• Privacy of Noisy Stochastic Gradient Descent: More Iterations without More Privacy Loss
• Differentially Private Federated Learning via Inexact ADMM with Multiple Local Updates