Subsampled Rényi Differential Privacy and Analytical Moments Accountant

Abstract

We study the problem of subsampling in differential privacy (DP), a question that is the centerpiece behind many successful differentially private machine learning algorithms. Specifically, we provide a tight upper bound on the Rényi Differential Privacy (RDP) (Mironov, 2017) parameters for algorithms that: (1) subsample the dataset, and then (2) applies a randomized mechanism M to the subsample, in terms of the RDP parameters of M and the subsampling probability parameter. Our results generalize the moments accounting technique, developed by Abadi et al. (2016) for the Gaussian mechanism, to any subsampled RDP mechanism.

Cited by in corpus (54)

• Privacy Amplification by Iteration
• Differentially Private Learning with Adaptive Clipping
• Differentially Private Learning Needs Better Features (or Much More Data)
• Understanding Gradient Clipping in Private SGD: A Geometric Perspective
• GS-WGAN: A Gradient-Sanitized Approach for Learning Differentially Private Generators
• Do Not Let Privacy Overbill Utility: Gradient Embedding Perturbation for Private Learning
• The Distributed Discrete Gaussian Mechanism for Federated Learning with Secure Aggregation
• DP-MERF: Differentially Private Mean Embeddings with Random Features for Practical Privacy-Preserving Data Generation
• Mitigating Membership Inference Attacks by Self-Distillation Through a Novel Ensemble Architecture
• Practical and Private (Deep) Learning without Sampling or Shuffling
• Improving Deep Learning with Differential Privacy using Gradient Encoding and Denoising
• DPIS: An Enhanced Mechanism for Differentially Private SGD with Importance Sampling
• DP-InstaHide: Provably Defusing Poisoning and Backdoor Attacks with Differentially Private Data Augmentations
• Training Production Language Models without Memorizing User Data
• Fast-adapting and Privacy-preserving Federated Recommender System
• Large Scale Private Learning via Low-rank Reparametrization
• Optimal Accounting of Differential Privacy via Characteristic Function
• RDP-GAN: A Rényi-Differential Privacy based Generative Adversarial Network
• Computing Tight Differential Privacy Guarantees Using FFT
• Efficient Privacy-Preserving Stochastic Nonconvex Optimization
• Differentially Private Federated Learning with Laplacian Smoothing
• DTGAN: Differential Private Training for Tabular GANs
• Federated Intrusion Detection for IoT with Heterogeneous Cohort Privacy
• Wide Network Learning with Differential Privacy
• Private Stochastic Non-Convex Optimization: Adaptive Algorithms and Tighter Generalization Bounds
• A(DP)${}^2$SGD: Asynchronous Decentralized Parallel Stochastic Gradient Descent with Differential Privacy
• Dynamic Differential-Privacy Preserving SGD
• Private and Communication-Efficient Edge Learning: A Sparse Differential Gaussian-Masking Distributed SGD Approach
• Renyi Differential Privacy of the Subsampled Shuffle Model in Distributed Learning
• PEARL: Data Synthesis via Private Embeddings and Adversarial Reconstruction Learning
• Differentially Private Deep Learning with Direct Feedback Alignment
• Differentially Private Bayesian Neural Networks on Accuracy, Privacy and Reliability
• Releasing Graph Neural Networks with Differential Privacy Guarantees
• DPlis: Boosting Utility of Differentially Private Deep Learning via Randomized Smoothing
• Computing Differential Privacy Guarantees for Heterogeneous Compositions Using FFT
• Tight Differential Privacy for Discrete-Valued Mechanisms and for the Subsampled Gaussian Mechanism Using FFT
• Effective and Privacy preserving Tabular Data Synthesizing
• Privacy Amplification by Mixing and Diffusion Mechanisms
• Revisiting Model-Agnostic Private Learning: Faster Rates and Active Learning
• Sensitivity analysis in differentially private machine learning using hybrid automatic differentiation
• Don't Generate Me: Training Differentially Private Generative Models with Sinkhorn Divergence
• Private Stochastic Convex Optimization: Efficient Algorithms for Non-smooth Objectives
• Differential Privacy Meets Federated Learning under Communication Constraints
• Task-aware Privacy Preservation for Multi-dimensional Data
• An Adaptive and Fast Convergent Approach to Differentially Private Deep Learning
• Combining Differential Privacy and Byzantine Resilience in Distributed SGD
• A closed form scale bound for the $(\epsilon ,\delta )$-differentially private Gaussian Mechanism valid for all privacy regimes
• Improving Differentially Private SGD via Randomly Sparsified Gradients
• Prior-Independent Auctions for the Demand Side of Federated Learning
• Privacy Amplification via Iteration for Shuffled and Online PNSGD
• Gradient-Leakage Resilient Federated Learning
• NeuralDP Differentially private neural networks by design
• Accuracy Gains from Privacy Amplification Through Sampling for Differential Privacy
• Amplifying Rényi Differential Privacy via Shuffling