FedPower: Privacy-Preserving Distributed Eigenspace Estimation
arXiv:2103.00704
Abstract
Eigenspace estimation is fundamental in machine learning and statistics, which has found applications in PCA, dimension reduction, and clustering, among others. The modern machine learning community usually assumes that data come from and belong to different organizations. The low communication power and the possible privacy breaches of data make the computation of eigenspace challenging. To address these challenges, we propose a class of algorithms called \textsf{FedPower} within the federated learning (FL) framework. \textsf{FedPower} leverages the well-known power method by alternating multiple local power iterations and a global aggregation step, thus improving communication efficiency. In the aggregation, we propose to weight each local eigenvector matrix with {\it Orthogonal Procrustes Transformation} (OPT) for better alignment. To ensure strong privacy protection, we add Gaussian noise in each iteration by adopting the notion of \emph{differential privacy} (DP). We provide convergence bounds for \textsf{FedPower} that are composed of different interpretable terms corresponding to the effects of Gaussian noise, parallelization, and random sampling of local machines. Additionally, we conduct experiments to demonstrate the effectiveness of our proposed algorithms.
References in corpus (13)
- Protection Against Reconstruction and Its Applications in Private Federated Learning
- Local SGD Converges Fast and Communicates Little
- Cooperative SGD: A unified Framework for the Design and Analysis of Communication-Efficient SGD Algorithms
- Gaussian Differential Privacy
- First Analysis of Local GD on Heterogeneous Data
- Federated Principal Component Analysis
- Communication-Efficient Local Decentralized SGD Methods
- Communication-efficient Algorithms for Distributed Stochastic Principal Component Analysis
- Distributed Stochastic Algorithms for High-rate Streaming Principal Component Analysis
- Randomized spectral co-clustering for large-scale directed networks
- Communication-Efficient Distributed SVD via Local Power Iterations
- Distributed Estimation for Principal Component Analysis: an Enlarged Eigenspace Analysis
- Delayed Projection Techniques for Linearly Constrained Problems: Convergence Rates, Acceleration, and Applications