An Efficient DP-SGD Mechanism for Large Scale NLP Models
arXiv:2107.14586
Abstract
Recent advances in deep learning have drastically improved performance on many Natural Language Understanding (NLU) tasks. However, the data used to train NLU models may contain private information such as addresses or phone numbers, particularly when drawn from human subjects. It is desirable that underlying models do not expose private information contained in the training data. Differentially Private Stochastic Gradient Descent (DP-SGD) has been proposed as a mechanism to build privacy-preserving models. However, DP-SGD can be prohibitively slow to train. In this work, we propose a more efficient DP-SGD for training using a GPU infrastructure and apply it to fine-tuning models based on LSTM and transformer architectures. We report faster training times, alongside accuracy, theoretical privacy guarantees and success of Membership inference attacks for our models and observe that fine-tuning with proposed variant of DP-SGD can yield competitive models without significant degradation in training time and improvement in privacy protection. We also make observations such as looser theoretical can translate into significant practical privacy gains.
References in corpus (12)
- Deep Learning with Differential Privacy
- MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
- Learning Differentially Private Recurrent Language Models
- Horovod: fast and easy distributed deep learning in TensorFlow
- Differentially Private Generative Adversarial Network
- Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfaces
- cpSGD: Communication-efficient and differentially-private distributed SGD
- Differentially Private Learning with Adaptive Clipping
- Towards Demystifying Membership Inference Attacks
- Auditing Differentially Private Machine Learning: How Private is Private SGD?
- LOGAN: Membership Inference Attacks Against Generative Models
- A(DP)SGD: Asynchronous Decentralized Parallel Stochastic Gradient Descent with Differential Privacy