Differentially Private Bayesian Neural Networks on Accuracy, Privacy and Reliability
arXiv:2107.08461
Abstract
Bayesian neural network (BNN) allows for uncertainty quantification in prediction, offering an advantage over regular neural networks that has not been explored in the differential privacy (DP) framework. We fill this important gap by leveraging recent development in Bayesian deep learning and privacy accounting to offer a more precise analysis of the trade-off between privacy and accuracy in BNN. We propose three DP-BNNs that characterize the weight uncertainty for the same network architecture in distinct ways, namely DP-SGLD (via the noisy gradient method), DP-BBP (via changing the parameters of interest) and DP-MC Dropout (via the model architecture). Interestingly, we show a new equivalence between DP-SGD and DP-SGLD, implying that some non-Bayesian DP training naturally allows for uncertainty quantification. However, the hyperparameters such as learning rate and batch size, can have different or even opposite effects in DP-SGD and DP-SGLD. Extensive experiments are conducted to compare DP-BNNs, in terms of privacy guarantee, prediction accuracy, uncertainty quantification, calibration, computation speed, and generalizability to network architecture. As a result, we observe a new tradeoff between the privacy and the reliability. When compared to non-DP and non-Bayesian approaches, DP-SGLD is remarkably accurate under strong privacy guarantee, demonstrating the great potential of DP-BNN in real-world tasks.
References in corpus (27)
- Deep Learning with Differential Privacy
- On Calibration of Modern Neural Networks
- Bayesian Convolutional Neural Networks with Bernoulli Approximate Variational Inference
- Stochastic Pooling for Regularization of Deep Convolutional Neural Networks
- Accurate Uncertainties for Deep Learning Using Calibrated Regression
- A generic framework for privacy preserving deep learning
- Differentially Private Generative Adversarial Network
- A Comprehensive guide to Bayesian Convolutional Neural Network with Variational Inference
- Preconditioned Stochastic Gradient Langevin Dynamics for Deep Neural Networks
- Privacy Amplification by Subsampling: Tight Analyses via Couplings and Divergences
- Gaussian Differential Privacy
- Privacy for Free: Posterior Sampling and Stochastic Gradient Monte Carlo
- Subsampled Rényi Differential Privacy and Analytical Moments Accountant
- Rényi Differential Privacy of the Sampled Gaussian Mechanism
- Revisiting the Calibration of Modern Neural Networks
- The Discrete Gaussian for Differential Privacy
- Calibration of Deep Probabilistic Models with Decoupled Bayesian Neural Networks
- Differentially Private Bayesian Learning on Distributed Data
- Efficient Per-Example Gradient Computations
- Differentially Private Bayesian Linear Regression
- Locally Private Bayesian Inference for Count Models
- Differentially Private Markov Chain Monte Carlo
- Wide Network Learning with Differential Privacy
- Fast and Memory Efficient Differentially Private-SGD via JL Projections
- Efficient Per-Example Gradient Computations in Convolutional Neural Networks
- Local Differential Privacy for Bayesian Optimization
- Differentially Private Dropout