Differentially Private Learning Needs Better Features (or Much More Data)
arXiv:2011.11660
Abstract
We demonstrate that differentially private machine learning has not yet reached its "AlexNet moment" on many canonical vision tasks: linear models trained on handcrafted features significantly outperform end-to-end deep neural networks for moderate privacy budgets. To exceed the performance of handcrafted features, we show that private learning requires either much more private data, or access to features learned on public data from a similar domain. Our work introduces simple yet strong baselines for differentially private learning that can inform the evaluation of future progress in this area.
ICLR 2021. Code available at https://github.com/ftramer/Handcrafted-DP
References in corpus (10)
- Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
- Semi-supervised Knowledge Transfer for Deep Learning from Private Training Data
- Enhanced Convolutional Neural Tangent Kernels
- Auditing Differentially Private Machine Learning: How Private is Private SGD?
- Large image datasets: A pyrrhic win for computer vision?
- Neural Kernels Without Tangents
- Tempered Sigmoid Activations for Deep Learning with Differential Privacy
- Improving Deep Learning with Differential Privacy using Gradient Encoding and Denoising
- Private Stochastic Non-Convex Optimization: Adaptive Algorithms and Tighter Generalization Bounds
- Stochastic Adaptive Line Search for Differentially Private Optimization