Natural Language Understanding with Privacy-Preserving BERT

Abstract

Privacy preservation remains a key challenge in data mining and Natural Language Understanding (NLU). Previous research shows that the input text or even text embeddings can leak private information. This concern motivates our research on effective privacy preservation approaches for pretrained Language Models (LMs). We investigate the privacy and utility implications of applying dx-privacy, a variant of Local Differential Privacy, to BERT fine-tuning in NLU applications. More importantly, we further propose privacy-adaptive LM pretraining methods and show that our approach can boost the utility of BERT dramatically while retaining the same level of privacy protection. We also quantify the level of privacy preservation and provide guidance on privacy configuration. Our experiments and findings lay the groundwork for future explorations of privacy-preserving NLU with pretrained LMs.

arXiv commentAccepted to CIKM 2021

References in corpus (13)

• Distributed Representations of Words and Phrases and their Compositionality
• Deep Learning with Differential Privacy
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Character-level Convolutional Networks for Text Classification
• Language Models are Few-Shot Learners
• Learning Differentially Private Recurrent Language Models
• GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
• Protection Against Reconstruction and Its Applications in Private Federated Learning
• Threats to Federated Learning: A Survey
• BERT with History Answer Embedding for Conversational Question Answering
• Generic Intent Representation in Web Search
• SynTF: Synthetic and Differentially Private Term Frequency Vectors for Privacy-Preserving Text Mining

Cited by in corpus (5)

• How to DP-fy ML: A Practical Guide to Machine Learning with Differential Privacy
• DP-Forward: Fine-tuning and Inference on Language Models with Differential Privacy in Forward Pass
• A Customized Text Sanitization Mechanism with Differential Privacy
• Complex QA and language models hybrid architectures, Survey
• Privacy-Preserving Parameter-Efficient Fine-Tuning for Large Language Model Services