Tempered Sigmoid Activations for Deep Learning with Differential Privacy

Abstract

Because learning sometimes involves sensitive data, machine learning algorithms have been extended to offer privacy for training data. In practice, this has been mostly an afterthought, with privacy-preserving models obtained by re-running training with a different optimizer, but using the model architectures that already performed well in a non-privacy-preserving setting. This approach leads to less than ideal privacy/utility tradeoffs, as we show here. Instead, we propose that model architectures are chosen ab initio explicitly for privacy-preserving training. To provide guarantees under the gold standard of differential privacy, one must bound as strictly as possible how individual training points can possibly affect model updates. In this paper, we are the first to observe that the choice of activation function is central to bounding the sensitivity of privacy-preserving deep learning. We demonstrate analytically and experimentally how a general family of bounded activation functions, the tempered sigmoids, consistently outperform unbounded activation functions like ReLU. Using this paradigm, we achieve new state-of-the-art accuracy on MNIST, FashionMNIST, and CIFAR10 without any modification of the learning procedure fundamentals or differential privacy analysis.

Cited by in corpus (20)

• Differentially Private Learning Needs Better Features (or Much More Data)
• Adaptive Machine Unlearning
• Mitigating Membership Inference Attacks by Self-Distillation Through a Novel Ensemble Architecture
• Practical and Private (Deep) Learning without Sampling or Shuffling
• Enabling Fast Differentially Private SGD via Just-in-Time Compilation and Vectorization
• Large Scale Private Learning via Low-rank Reparametrization
• Evading Curse of Dimensionality in Unconstrained Private GLMs via Private Gradient Descent
• On the Convergence and Calibration of Deep Learning with Differential Privacy
• Deep Learning with Label Differential Privacy
• Complex-valued Federated Learning with Differential Privacy and MRI Applications
• Hyperparameter Tuning with Renyi Differential Privacy
• Renyi Differential Privacy of the Subsampled Shuffle Model in Distributed Learning
• Public Data-Assisted Mirror Descent for Private Model Training
• DPlis: Boosting Utility of Differentially Private Deep Learning via Randomized Smoothing
• Sensitivity analysis in differentially private machine learning using hybrid automatic differentiation
• An automatic differentiation system for the age of differential privacy
• Selective Differential Privacy for Language Modeling
• Architecture Matters: Investigating the Influence of Differential Privacy on Neural Network Design
• Efficient Hyperparameter Optimization for Differentially Private Deep Learning
• Improving Differentially Private SGD via Randomly Sparsified Gradients