Benchmarking Differential Privacy and Federated Learning for BERT Models
arXiv:2106.13973
Abstract
Natural Language Processing (NLP) techniques can be applied to help with the diagnosis of medical conditions such as depression, using a collection of a person's utterances. Depression is a serious medical illness that can have adverse effects on how one feels, thinks, and acts, which can lead to emotional and physical problems. Due to the sensitive nature of such data, privacy measures need to be taken for handling and training models with such data. In this work, we study the effects that the application of Differential Privacy (DP) has, in both a centralized and a Federated Learning (FL) setup, on training contextualized language models (BERT, ALBERT, RoBERTa and DistilBERT). We offer insights on how to privately train NLP models and what architectures and setups provide more desirable privacy utility trade-offs. We envisage this work to be used in future healthcare and mental health studies to keep medical history private. Therefore, we provide an open-source implementation of this work.
arXiv comment4 pages, 3 tables, 1 figure
References in corpus (12)
- Deep Learning with Differential Privacy
- Communication-Efficient Learning of Deep Networks from Decentralized Data
- Federated Learning with Non-IID Data
- A Survey on Federated Learning Systems: Vision, Hype and Reality for Data Privacy and Protection
- Learning Differentially Private Recurrent Language Models
- Federated Machine Learning: Concept and Applications
- Differentially Private Federated Learning: A Client Level Perspective
- Federated Meta-Learning with Fast Convergence and Efficient Communication
- Federated Multi-Task Learning
- Differentially Private Learning Needs Better Features (or Much More Data)
- Towards Robust and Privacy-preserving Text Representations
- Privacy Regularization: Joint Privacy-Utility Optimization in Language Models