Learning Privately over Distributed Features: An ADMM Sharing Approach

Abstract

Distributed machine learning has been widely studied in order to handle exploding amount of data. In this paper, we study an important yet less visited distributed learning problem where features are inherently distributed or vertically partitioned among multiple parties, and sharing of raw data or model parameters among parties is prohibited due to privacy concerns. We propose an ADMM sharing framework to approach risk minimization over distributed features, where each party only needs to share a single value for each sample in the training process, thus minimizing the data leakage risk. We establish convergence and iteration complexity results for the proposed parallel ADMM algorithm under non-convex loss. We further introduce a novel differentially private ADMM sharing algorithm and bound the privacy guarantee with carefully designed noise perturbation. The experiments based on a prototype system shows that the proposed ADMM algorithms converge efficiently in a robust fashion, demonstrating advantage over gradient based methods especially for data set with high dimensional feature spaces.

References in corpus (4)

• Deep Learning with Differential Privacy
• On the Linear Convergence of the ADMM in Decentralized Consensus Optimization
• Privacy via the Johnson-Lindenstrauss Transform
• Supervised Learning Under Distributed Features

Cited by in corpus (10)

• A Communication Efficient Collaborative Learning Framework for Distributed Features
• Backdoor attacks and defenses in feature-partitioned collaborative learning
• Practical Vertical Federated Learning with Unsupervised Representation Learning
• Differentially Private Vertical Federated Clustering
• Self-supervised Cross-silo Federated Neural Architecture Search
• Differentially Private ADMM for Convex Distributed Learning: Improved Accuracy via Multi-Step Approximation
• Privacy-Preserving Distributed Zeroth-Order Optimization
• The Internet of Things as a Deep Neural Network
• From Noisy Fixed-Point Iterations to Private ADMM for Centralized and Federated Learning
• Decentralized Non-Convex Learning with Linearly Coupled Constraints