Differentially-Private "Draw and Discard" Machine Learning

Abstract

In this work, we propose a novel framework for privacy-preserving client-distributed machine learning. It is motivated by the desire to achieve differential privacy guarantees in the local model of privacy in a way that satisfies all systems constraints using asynchronous client-server communication and provides attractive model learning properties. We call it "Draw and Discard" because it relies on random sampling of models for load distribution (scalability), which also provides additional server-side privacy protections and improved model quality through averaging. We present the mechanics of client and server components of "Draw and Discard" and demonstrate how the framework can be applied to learning Generalized Linear models. We then analyze the privacy guarantees provided by our approach against several types of adversaries and showcase experimental results that provide evidence for the framework's viability in practical deployments.

References in corpus (11)

• Deep Learning with Differential Privacy
• Renyi Differential Privacy
• Learning Differentially Private Recurrent Language Models
• Revisiting Small Batch Training for Deep Neural Networks
• Scalable Private Learning with PATE
• Privacy Loss in Apple's Implementation of Differential Privacy on MacOS 10.12
• Privacy via the Johnson-Lindenstrauss Transform
• Concentrated Differential Privacy: Simplifications, Extensions, and Lower Bounds
• Practical Locally Private Heavy Hitters
• Privacy in Information-Rich Intelligent Infrastructure
• Heavy Hitters and the Structure of Local Privacy

Cited by in corpus (11)

• Towards Federated Learning at Scale: System Design
• LEAF: A Benchmark for Federated Settings
• Orthogonal Gradient Descent for Continual Learning
• On the Practicality of Differential Privacy in Federated Learning by Tuning Iteration Times
• PRECAD: Privacy-Preserving and Robust Federated Learning via Crypto-Aided Differential Privacy
• SoK: Training Machine Learning Models over Multiple Sources with Privacy Preservation
• Privacy-Preserving Image Classification in the Local Setting
• Differentially Private Bayesian Inference for Generalized Linear Models
• DP-REC: Private & Communication-Efficient Federated Learning
• Optimizing the Numbers of Queries and Replies in Federated Learning with Differential Privacy
• Privacy-Preserving Bandits