Differentially Private Deep Learning with Direct Feedback Alignment
arXiv:2010.03701
Abstract
Standard methods for differentially private training of deep neural networks replace back-propagated mini-batch gradients with biased and noisy approximations to the gradient. These modifications to training often result in a privacy-preserving model that is significantly less accurate than its non-private counterpart. We hypothesize that alternative training algorithms may be more amenable to differential privacy. Specifically, we examine the suitability of direct feedback alignment (DFA). We propose the first differentially private method for training deep neural networks with DFA and show that it achieves significant gains in accuracy (often by 10-20%) compared to backprop-based differentially private training on a variety of architectures (fully connected, convolutional) and datasets.
References in corpus (13)
- Deep Learning with Differential Privacy
- Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
- Renyi Differential Privacy
- Learning Differentially Private Recurrent Language Models
- Differentially Private Generative Adversarial Network
- Assessing the Scalability of Biologically-Motivated Deep Learning Algorithms and Architectures
- Subsampled Rényi Differential Privacy and Analytical Moments Accountant
- Differentially Private Learning with Adaptive Clipping
- Deep Learning without Weight Transport
- How Important is Weight Symmetry in Backpropagation?
- Feedback alignment in deep convolutional networks
- Differentially Private Data Generative Models
- Removing Disparate Impact of Differentially Private Stochastic Gradient Descent on Model Accuracy