Analyzing Information Leakage of Updates to Natural Language Models

Abstract

To continuously improve quality and reflect changes in data, machine learning applications have to regularly retrain and update their core models. We show that a differential analysis of language model snapshots before and after an update can reveal a surprising amount of detailed information about changes in the training data. We propose two new metrics---\emph{differential score} and \emph{differential rank}---for analyzing the leakage due to updates of natural language models. We perform leakage analysis using these metrics across models trained on several different datasets using different methods and configurations. We discuss the privacy implications of our findings, propose mitigation strategies and evaluate their effect.

References in corpus (7)

• Deep Learning with Differential Privacy
• Recurrent Neural Network Regularization
• Learning Differentially Private Recurrent Language Models
• The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks
• Pointer Sentinel Mixture Models
• Deep Models Under the GAN: Information Leakage from Collaborative Deep Learning
• Updates-Leak: Data Set Inference and Reconstruction Attacks in Online Learning

Cited by in corpus (20)

• Extracting Training Data from Large Language Models
• When Machine Unlearning Jeopardizes Privacy
• Privacy in Deep Learning: A Survey
• Do Language Models Plagiarize?
• DP-Forward: Fine-tuning and Inference on Language Models with Differential Privacy in Forward Pass
• AI Safety for Everyone
• Leakage of Dataset Properties in Multi-Party Machine Learning
• Your Code Secret Belongs to Me: Neural Code Completion Tools Can Memorize Hard-Coded Credentials
• Training Data Leakage Analysis in Language Models
• Membership Inference Attacks and Generalization: A Causal Perspective
• SoK: The Privacy Paradox of Large Language Models: Advancements, Privacy Risks, and Mitigation
• Membership Inference on Word Embedding and Beyond
• LMSanitator: Defending Prompt-Tuning Against Task-Agnostic Backdoors
• SoK: Machine Learning Governance
• Quantifying and Mitigating Privacy Risks of Contrastive Learning
• Hidden Backdoors in Human-Centric Language Models
• Evaluating Language Model Agency through Negotiations
• Information Leakage from Data Updates in Machine Learning Models
• The Connection between Out-of-Distribution Generalization and Privacy of ML Models
• Memorization of Named Entities in Fine-tuned BERT Models