DP-Forward: Fine-tuning and Inference on Language Models with Differential Privacy in Forward Pass

Abstract

Differentially private stochastic gradient descent (DP-SGD) adds noise to gradients in back-propagation, safeguarding training data from privacy leakage, particularly membership inference. It fails to cover (inference-time) threats like embedding inversion and sensitive attribute inference. It is also costly in storage and computation when used to fine-tune large pre-trained language models (LMs). We propose DP-Forward, which directly perturbs embedding matrices in the forward pass of LMs. It satisfies stringent local DP requirements for training and inference data. To instantiate it using the smallest matrix-valued noise, we devise an analytic matrix Gaussian~mechanism (aMGM) by drawing possibly non-i.i.d. noise from a matrix Gaussian distribution. We then investigate perturbing outputs from different hidden (sub-)layers of LMs with aMGM noises. Its utility on three typical tasks almost hits the non-private baseline and outperforms DP-SGD by up to 7.7pp at a moderate privacy level. It saves 3$\times$ time and memory costs compared to DP-SGD with the latest high-speed library. It also reduces the average success rates of embedding inversion and sensitive attribute inference by up to 88pp and 41pp, respectively, whereas DP-SGD fails.

arXiv commentTo appear at ACM CCS '23. This is the full version. The first two authors contribute equally

References in corpus (21)

• Deep Learning with Differential Privacy
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• LoRA: Low-Rank Adaptation of Large Language Models
• A Survey on Deep Learning for Named Entity Recognition
• Renyi Differential Privacy
• Learning Differentially Private Recurrent Language Models
• Differentially Private Empirical Risk Minimization
• The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks
• Analyzing Information Leakage of Updates to Natural Language Models
• Systematic Evaluation of Privacy Risks of Machine Learning Models
• Large Language Models Can Be Strong Differentially Private Learners
• Differentially Private Fine-tuning of Language Models
• Natural Language Understanding with Privacy-Preserving BERT
• MVG Mechanism: Differential Privacy under Matrix-Valued Query
• The Lipschitz Constant of Self-Attention
• SynTF: Synthetic and Differentially Private Term Frequency Vectors for Privacy-Preserving Text Mining
• Large Scale Private Learning via Low-rank Reparametrization
• Lipschitz Normalization for Self-Attention Layers with Application to Graph Neural Networks
• Does Label Differential Privacy Prevent Label Inference Attacks?
• Improved Matrix Gaussian Mechanism for Differential Privacy
• Locally Differentially Private Sparse Vector Aggregation

Cited by in corpus (2)

• Unveiling the Vulnerability of Private Fine-Tuning in Split-Based Frameworks for Large Language Models: A Bidirectionally Enhanced Attack
• VFLAIR-LLM: A Comprehensive Framework and Benchmark for Split Learning of LLMs