Training Data Leakage Analysis in Language Models

Abstract

Recent advances in neural network based language models lead to successful deployments of such models, improving user experience in various applications. It has been demonstrated that strong performance of language models comes along with the ability to memorize rare training samples, which poses serious privacy threats in case the model is trained on confidential user content. In this work, we introduce a methodology that investigates identifying the user content in the training data that could be leaked under a strong and realistic threat model. We propose two metrics to quantify user-level data leakage by measuring a model's ability to produce unique sentence fragments within training data. Our metrics further enable comparing different models trained on the same data in terms of privacy. We demonstrate our approach through extensive numerical studies on both RNN and Transformer based models. We further illustrate how the proposed metrics can be utilized to investigate the efficacy of mitigations like differentially private training or API hardening.

References in corpus (17)

• Deep Learning with Differential Privacy
• Power laws, Pareto distributions and Zipf's law
• Language Models are Few-Shot Learners
• Comprehensive Privacy Analysis of Deep Learning: Passive and Active White-box Inference Attacks against Centralized and Federated Learning
• ERNIE: Enhanced Representation through Knowledge Integration
• Learning Differentially Private Recurrent Language Models
• The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks
• Extracting Training Data from Large Language Models
• Understanding Membership Inferences on Well-Generalized Learning Models
• White-box vs Black-box: Bayes Optimal Strategies for Membership Inference
• Stolen Memories: Leveraging Model Memorization for Calibrated White-Box Membership Inference
• Label-Only Membership Inference Attacks
• Analyzing Information Leakage of Updates to Natural Language Models
• Towards Demystifying Membership Inference Attacks
• Differentially Private Learning Needs Better Features (or Much More Data)
• Conversational Contextual Cues: The Case of Personalization and History for Response Ranking
• ML Privacy Meter: Aiding Regulatory Compliance by Quantifying the Privacy Risks of Machine Learning

Cited by in corpus (3)

• An Evaluation on Large Language Model Outputs: Discourse and Memorization
• Membership Inference on Word Embedding and Beyond
• Privacy Regularization: Joint Privacy-Utility Optimization in Language Models