Extracting Training Data from Large Language Models

Abstract

It has become common to publish large (billion parameter) language models that have been trained on private datasets. This paper demonstrates that in such settings, an adversary can perform a training data extraction attack to recover individual training examples by querying the language model. We demonstrate our attack on GPT-2, a language model trained on scrapes of the public Internet, and are able to extract hundreds of verbatim text sequences from the model's training data. These extracted examples include (public) personally identifiable information (names, phone numbers, and email addresses), IRC conversations, code, and 128-bit UUIDs. Our attack is possible even though each of the above sequences are included in just one document in the training data. We comprehensively evaluate our extraction attack to understand the factors that contribute to its success. Worryingly, we find that larger models are more vulnerable than smaller models. We conclude by drawing lessons and discussing possible safeguards for training large language models.

References in corpus (16)

• Deep Learning with Differential Privacy
• Language Models are Few-Shot Learners
• Comprehensive Privacy Analysis of Deep Learning: Passive and Active White-box Inference Attacks against Centralized and Federated Learning
• Scaling Laws for Neural Language Models
• Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
• Learning Differentially Private Recurrent Language Models
• The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks
• Towards a Human-like Open-Domain Chatbot
• ReCon: Revealing and Controlling PII Leaks in Mobile Network Traffic
• Understanding Membership Inferences on Well-Generalized Learning Models
• Language Models as Knowledge Bases?
• Analyzing Information Leakage of Updates to Natural Language Models
• Gmail Smart Compose: Real-Time Assisted Writing
• Efficient Adaptation of Pretrained Transformers for Abstractive Summarization
• Understanding Unintended Memorization in Federated Learning
• Training Production Language Models without Memorizing User Data

Cited by in corpus (39)

• When Large Language Models Meet Personalization: Perspectives of Challenges and Opportunities
• Auditing large language models: a three-layered approach
• Let's have a chat! A Conversation with ChatGPT: Technology, Applications, and Limitations
• Large language models surpass human experts in predicting neuroscience results
• "It's a Fair Game", or Is It? Examining How Users Navigate Disclosure Risks and Benefits When Using LLM-Based Conversational Agents
• Trustworthy AI: Securing Sensitive Data in Large Language Models
• KLUE: Korean Language Understanding Evaluation
• Black-Box Access is Insufficient for Rigorous AI Audits
• Differentially Private Learning Needs Better Features (or Much More Data)
• HILL: A Hallucination Identifier for Large Language Models
• Do Language Models Plagiarize?
• Red Alarm for Pre-trained Models: Universal Vulnerability to Neuron-Level Backdoor Attacks
• AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing
• Wordcraft: a Human-AI Collaborative Editor for Story Writing
• An Evaluation on Large Language Model Outputs: Discourse and Memorization
• Program Synthesis with Large Language Models
• Building Privacy-Preserving and Secure Geospatial Artificial Intelligence Foundation Models
• Training Data Leakage Analysis in Language Models
• What's in the Box? A Preliminary Analysis of Undesirable Content in the Common Crawl Corpus
• DPIS: An Enhanced Mechanism for Differentially Private SGD with Importance Sampling
• Application of AI-based Models for Online Fraud Detection and Analysis
• Membership Inference on Word Embedding and Beyond
• Sneaky Spikes: Uncovering Stealthy Backdoor Attacks in Spiking Neural Networks with Neuromorphic Data
• Covariance-Aware Private Mean Estimation Without Private Covariance Estimation
• The Sensory Neuron as a Transformer: Permutation-Invariant Neural Networks for Reinforcement Learning
• Web Archives Metadata Generation with GPT-4o: Challenges and Insights
• SLMIA-SR: Speaker-Level Membership Inference Attacks against Speaker Recognition Systems
• Hyperparameter Tuning with Renyi Differential Privacy
• Trained Without My Consent: Detecting Code Inclusion In Language Models Trained on Code
• Where have you been? A Study of Privacy Risk for Point-of-Interest Recommendation
• PATE-AAE: Incorporating Adversarial Autoencoder into Private Aggregation of Teacher Ensembles for Spoken Command Classification
• Defending Medical Image Diagnostics against Privacy Attacks using Generative Methods
• LazyDP: Co-Designing Algorithm-Software for Scalable Training of Differentially Private Recommendation Models
• Correlation inference attacks against machine learning models
• Benchmarking Multimodal AutoML for Tabular Data with Text Fields
• Differential Privacy for Text Analytics via Natural Text Sanitization
• Nob-MIAs: Non-biased Membership Inference Attacks Assessment on Large Language Models with Ex-Post Dataset Construction
• Memorization of Named Entities in Fine-tuned BERT Models
• Several Issues Regarding Data Governance in AGI