Architecture Matters: Investigating the Influence of Differential Privacy on Neural Network Design
arXiv:2111.14924
Abstract
One barrier to more widespread adoption of differentially private neural networks is the entailed accuracy loss. To address this issue, the relationship between neural network architectures and model accuracy under differential privacy constraints needs to be better understood. As a first step, we test whether extant knowledge on architecture design also holds in the differentially private setting. Our findings show that it does not; architectures that perform well without differential privacy, do not necessarily do so with differential privacy. Consequently, extant knowledge on neural network architecture design cannot be seamlessly translated into the differential privacy context. Future research is required to better understand the relationship between neural network architectures and model accuracy to enable better architecture design choices under differential privacy constraints.
arXiv commentTo be presented at Privacy in Machine Learning (NeurIPS 2021 Workshop)
References in corpus (7)
- Deep Learning with Differential Privacy
- Communication-Efficient Learning of Deep Networks from Decentralized Data
- Neural Architecture Search: A Survey
- Multi-digit Number Recognition from Street View Imagery using Deep Convolutional Neural Networks
- Why gradient clipping accelerates training: A theoretical justification for adaptivity
- Tempered Sigmoid Activations for Deep Learning with Differential Privacy
- Three Tools for Practical Differential Privacy