Differentially private training of neural networks with Langevin dynamics for calibrated predictive uncertainty

Abstract

We show that differentially private stochastic gradient descent (DP-SGD) can yield poorly calibrated, overconfident deep learning models. This represents a serious issue for safety-critical applications, e.g. in medical diagnosis. We highlight and exploit parallels between stochastic gradient Langevin dynamics, a scalable Bayesian inference technique for training deep neural networks, and DP-SGD, in order to train differentially private, Bayesian neural networks with minor adjustments to the original (DP-SGD) algorithm. Our approach provides considerably more reliable uncertainty estimates than DP-SGD, as demonstrated empirically by a reduction in expected calibration error (MNIST $\sim 5$-fold, Pediatric Pneumonia Dataset $\sim 2$-fold).

arXiv commentAccepted to the ICML 2021 Theory and Practice of Differential Privacy Workshop

References in corpus (5)

• Deep Learning with Differential Privacy
• Predicting Cardiovascular Risk Factors from Retinal Fundus Photographs using Deep Learning
• Privacy for Free: Posterior Sampling and Stochastic Gradient Monte Carlo
• The Case for Bayesian Deep Learning
• Stochastic Gradient Langevin Dynamics Algorithms with Adaptive Drifts