The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks

Abstract

This paper describes a testing methodology for quantitatively assessing the risk that rare or unique training-data sequences are unintentionally memorized by generative sequence models---a common type of machine-learning model. Because such models are sometimes trained on sensitive data (e.g., the text of users' private messages), this methodology can benefit privacy by allowing deep-learning practitioners to select means of training that minimize such memorization. In experiments, we show that unintended memorization is a persistent, hard-to-avoid issue that can have serious consequences. Specifically, for models trained without consideration of memorization, we describe new, efficient procedures that can extract unique, secret sequences, such as credit card numbers. We show that our testing strategy is a practical and easy-to-use first line of defense, e.g., by describing its application to quantitatively limit data exposure in Google's Smart Compose, a commercial text-completion neural network trained on millions of users' email messages.

Cited by in corpus (143)

• Comprehensive Privacy Analysis of Deep Learning: Passive and Active White-box Inference Attacks against Centralized and Federated Learning
• How To Backdoor Federated Learning
• GAN-Leaks: A Taxonomy of Membership Inference Attacks against Generative Models
• Extracting Training Data from Large Language Models
• A Survey of Privacy Attacks in Machine Learning
• Using GANs for Sharing Networked Time Series Data: Challenges, Initial Promise, and Open Questions
• How to DP-fy ML: A Practical Guide to Machine Learning with Differential Privacy
• Zero-Shot Machine Unlearning
• RobustBench: a standardized adversarial robustness benchmark
• FastSecAgg: Scalable Secure Aggregation for Privacy-Preserving Federated Learning
• Thieves on Sesame Street! Model Extraction of BERT-based APIs
• Label-Only Membership Inference Attacks
• Analyzing Information Leakage of Updates to Natural Language Models
• Evaluating Differentially Private Machine Learning in Practice
• Systematic Evaluation of Privacy Risks of Machine Learning Models
• Towards Demystifying Membership Inference Attacks
• Backdoor Pre-trained Models Can Transfer to All
• Large Language Models Can Be Strong Differentially Private Learners
• On the Effectiveness of Mitigating Data Poisoning Attacks with Gradient Shaping
• Remember What You Want to Forget: Algorithms for Machine Unlearning
• A Study of Face Obfuscation in ImageNet
• Differentially Private Learning Needs Better Features (or Much More Data)
• The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset
• Node-Level Membership Inference Attacks Against Graph Neural Networks
• DP-Forward: Fine-tuning and Inference on Language Models with Differential Privacy in Forward Pass
• Differentially Private Fine-tuning of Language Models
• DataLens: Scalable Privacy Preserving Training via Gradient Compression and Aggregation
• ML-Doctor: Holistic Risk Assessment of Inference Attacks Against Machine Learning Models
• Patchwork Learning: A Paradigm Towards Integrative Analysis across Diverse Biomedical Data Sources
• Dopamine: Differentially Private Federated Learning on Medical Data
• Modifying Memories in Transformer Models
• Privacy for Free: Communication-Efficient Learning with Differential Privacy Using Sketches
• Stealing Links from Graph Neural Networks
• ML Privacy Meter: Aiding Regulatory Compliance by Quantifying the Privacy Risks of Machine Learning
• Approximate Data Deletion from Machine Learning Models
• Privacy Accounting and Quality Control in the Sage Differentially Private ML Platform
• Towards Probabilistic Verification of Machine Unlearning
• A Customized Text Sanitization Mechanism with Differential Privacy
• Leakage of Dataset Properties in Multi-Party Machine Learning
• Debiasing Learning for Membership Inference Attacks Against Recommender Systems
• Do Not Let Privacy Overbill Utility: Gradient Embedding Perturbation for Private Learning
• The Distributed Discrete Gaussian Mechanism for Federated Learning with Secure Aggregation
• Differential Privacy in Natural Language Processing: The Story So Far
• Enhancing the Privacy of Federated Learning with Sketching
• Machine Learning with Membership Privacy using Adversarial Regularization
• Mitigating Membership Inference Attacks by Self-Distillation Through a Novel Ensemble Architecture
• Training Data Leakage Analysis in Language Models
• Poisoning the Unlabeled Dataset of Semi-Supervised Learning
• Evaluating Privacy-Preserving Machine Learning in Critical Infrastructures: A Case Study on Time-Series Classification
• Cloudy with high chance of DBMS: A 10-year prediction for Enterprise-Grade ML
• Fidelity and Privacy of Synthetic Medical Data
• Enabling Fast Differentially Private SGD via Just-in-Time Compilation and Vectorization
• Privacy attacks for automatic speech recognition acoustic models in a federated learning framework
• Nine Potential Pitfalls when Designing Human-AI Co-Creative Systems
• Flexible Dataset Distillation: Learn Labels Instead of Images
• Knowledge Cross-Distillation for Membership Privacy
• Adversarial Training Versus Weight Decay
• Membership Inference Attacks and Generalization: A Causal Perspective
• Anticipating Safety Issues in E2E Conversational AI: Framework and Tooling
• Auditing Data Provenance in Text-Generation Models
• Measuring Data Leakage in Machine-Learning Models with Fisher Information
• This Person (Probably) Exists. Identity Membership Attacks Against GAN Generated Faces
• Medical Image Synthesis for Data Augmentation and Anonymization using Generative Adversarial Networks
• Self-Damaging Contrastive Learning
• Membership Inference on Word Embedding and Beyond
• Large Scale Private Learning via Low-rank Reparametrization
• Enhanced Membership Inference Attacks against Machine Learning Models
• LOGAN: Membership Inference Attacks Against Generative Models
• Mitigating the OWASP Top 10 For Large Language Models Applications using Intelligent Agents
• AUTOLYCUS: Exploiting Explainable AI (XAI) for Model Extraction Attacks against Interpretable Models
• Free-rider Attacks on Model Aggregation in Federated Learning
• Information Leakage in Embedding Models
• Helen: Maliciously Secure Coopetitive Learning for Linear Models
• SoK: Machine Learning Governance
• Efficient Privacy-Preserving Stochastic Nonconvex Optimization
• Covariance-Aware Private Mean Estimation Without Private Covariance Estimation
• FedSKETCH: Communication-Efficient and Private Federated Learning via Sketching
• On the Practicality of Differential Privacy in Federated Learning by Tuning Iteration Times
• Differentially Private Multivariate Time Series Forecasting of Aggregated Human Mobility With Deep Learning: Input or Gradient Perturbation?
• A Marauder's Map of Security and Privacy in Machine Learning
• Future Lens: Anticipating Subsequent Tokens from a Single Hidden State
• Robin Hood and Matthew Effects: Differential Privacy Has Disparate Impact on Synthetic Data
• Déjà Vu: an empirical evaluation of the memorization properties of ConvNets
• Wide Network Learning with Differential Privacy
• CaPC Learning: Confidential and Private Collaborative Learning
• Weighted Distributed Differential Privacy ERM: Convex and Non-convex
• Antipodes of Label Differential Privacy: PATE and ALIBI
• On the Convergence and Calibration of Deep Learning with Differential Privacy
• Fast and Memory Efficient Differentially Private-SGD via JL Projections
• Revocable Federated Learning: A Benchmark of Federated Forest
• Membership Inference Attacks Against Recommender Systems
• FaceLeaks: Inference Attacks against Transfer Learning Models via Black-box Queries
• CryptoSPN: Privacy-preserving Sum-Product Network Inference
• Privacy-preserving Federated Bayesian Learning of a Generative Model for Imbalanced Classification of Clinical Data
• PAC learning with stable and private predictions
• DeepObliviate: A Powerful Charm for Erasing Data Residual Memory in Deep Neural Networks
• On the Importance of Difficulty Calibration in Membership Inference Attacks
• Actor Critic with Differentially Private Critic
• Report from the NSF Future Directions Workshop, Toward User-Oriented Agents: Research Directions and Challenges
• Voting-based Approaches For Differentially Private Federated Learning
• PreCog: Exploring the Relation between Memorization and Performance in Pre-trained Language Models
• Differentially Private SGD with Non-Smooth Losses
• Privacy Regularization: Joint Privacy-Utility Optimization in Language Models
• A Lightweight Privacy-Preserving Scheme Using Label-based Pixel Block Mixing for Image Classification in Deep Learning
• Generalization Techniques Empirically Outperform Differential Privacy against Membership Inference
• Robustness Threats of Differential Privacy
• Source Inference Attacks in Federated Learning
• Model Extraction Attacks against Recurrent Neural Networks
• An Algorithmic Framework For Differentially Private Data Analysis on Trusted Processors
• Private Speech Classification with Secure Multiparty Computation
• DPlis: Boosting Utility of Differentially Private Deep Learning via Randomized Smoothing
• Privacy in Practice: Private COVID-19 Detection in X-Ray Images (Extended Version)
• The Connection between Out-of-Distribution Generalization and Privacy of ML Models
• Have you forgotten? A method to assess if machine learning models have forgotten data
• Abstraction and Symbolic Execution of Deep Neural Networks with Bayesian Approximation of Hidden Features
• The Trade-Offs of Private Prediction
• Green Lighting ML: Confidentiality, Integrity, and Availability of Machine Learning Systems in Deployment
• Node-Level Differentially Private Graph Neural Networks
• Machine unlearning via GAN
• De-identification of Privacy-related Entities in Job Postings
• Alleviating Privacy Attacks via Causal Learning
• Fundamental Limits of Membership Inference Attacks on Machine Learning Models
• GRAFFL: Gradient-free Federated Learning of a Bayesian Generative Model
• The Cost of a Reductions Approach to Private Fair Optimization
• An Analysis Of Protected Health Information Leakage In Deep-Learning Based De-Identification Algorithms
• On the Difficulty of Membership Inference Attacks
• The Influence of Dropout on Membership Inference in Differentially Private Models
• EMA: Auditing Data Removal from Trained Models
• TableGAN-MCA: Evaluating Membership Collisions of GAN-Synthesized Tabular Data Releasing
• Optimizing the Numbers of Queries and Replies in Federated Learning with Differential Privacy
• Self-supervised denoising for massive noisy images
• Learning to Collaborate for User-Controlled Privacy
• AMI-FML: A Privacy-Preserving Federated Machine Learning Framework for AMI
• CAPE: Context-Aware Private Embeddings for Private Language Learning
• Differentially Private n-gram Extraction
• Selective Differential Privacy for Language Modeling
• Generating synthetic transactional profiles
• STAN: Synthetic Network Traffic Generation with Generative Neural Models
• DP-SGD vs PATE: Which Has Less Disparate Impact on GANs?
• A Comparison of Methods for Neural Network Aggregation
• Privacy-Preserving Portrait Matting
• Human-Imitating Metrics for Training and Evaluating Privacy Preserving Emotion Recognition Models Using Sociolinguistic Knowledge
• TEDL: A Text Encryption Method Based on Deep Learning