Privacy for Free: Communication-Efficient Learning with Differential Privacy Using Sketches

Abstract

Communication and privacy are two critical concerns in distributed learning. Many existing works treat these concerns separately. In this work, we argue that a natural connection exists between methods for communication reduction and privacy preservation in the context of distributed machine learning. In particular, we prove that Count Sketch, a simple method for data stream summarization, has inherent differential privacy properties. Using these derived privacy guarantees, we propose a novel sketch-based framework (DiffSketch) for distributed learning, where we compress the transmitted messages via sketches to simultaneously achieve communication efficiency and provable privacy benefits. Our evaluation demonstrates that DiffSketch can provide strong differential privacy guarantees (e.g., $\epsilon$= 1) and reduce communication by 20-50x with only marginal decreases in accuracy. Compared to baselines that treat privacy and communication separately, DiffSketch improves absolute test accuracy by 5%-50% while offering the same privacy guarantees and communication compression.

References in corpus (16)

• Deep Learning with Differential Privacy
• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Federated Learning: Strategies for Improving Communication Efficiency
• Learning Differentially Private Recurrent Language Models
• The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks
• Federated Optimization in Heterogeneous Networks
• LEAF: A Benchmark for Federated Settings
• Expanding the Reach of Federated Learning by Reducing Client Resource Requirements
• AIDE: Fast and Communication Efficient Distributed Optimization
• Scalable and Differentially Private Distributed Aggregation in the Shuffled Model
• Secure Computation for Machine Learning With SPDZ
• Enhancing the Privacy of Federated Learning with Sketching
• Differentially Private Meta-Learning
• Federated Heavy Hitters Discovery with Differential Privacy
• Total stochastic gradient algorithms and applications in reinforcement learning
• Gradient Distribution Priors for Biomedical Image Processing

Cited by in corpus (8)

• Privacy-Preserving Machine Learning: Methods, Challenges and Directions
• FetchSGD: Communication-Efficient Federated Learning with Sketching
• Local Differential Privacy based Federated Learning for Internet of Things
• Enhancing the Privacy of Federated Learning with Sketching
• User-Level Privacy-Preserving Federated Learning: Analysis and Performance Optimization
• FedSKETCH: Communication-Efficient and Private Federated Learning via Sketching
• Trends and Advancements in Deep Neural Network Communication
• Matrix Sketching for Secure Collaborative Machine Learning