Understanding Membership Inferences on Well-Generalized Learning Models

Abstract

Membership Inference Attack (MIA) determines the presence of a record in a machine learning model's training data by querying the model. Prior work has shown that the attack is feasible when the model is overfitted to its training data or when the adversary controls the training algorithm. However, when the model is not overfitted and the adversary does not control the training algorithm, the threat is not well understood. In this paper, we report a study that discovers overfitting to be a sufficient but not a necessary condition for an MIA to succeed. More specifically, we demonstrate that even a well-generalized model contains vulnerable instances subject to a new generalized MIA (GMIA). In GMIA, we use novel techniques for selecting vulnerable instances and detecting their subtle influences ignored by overfitting metrics. Specifically, we successfully identify individual records with high precision in real-world datasets by querying black-box machine learning models. Further we show that a vulnerable record can even be indirectly attacked by querying other related records and existing generalization techniques are found to be less effective in protecting the vulnerable instances. Our findings sharpen the understanding of the fundamental cause of the problem: the unique influences the training instance may have on the model.

References in corpus (12)

• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
• Deep Learning with Differential Privacy
• Delving into Transferable Adversarial Examples and Black-box Attacks
• Learning Differentially Private Recurrent Language Models
• Robust Physical-World Attacks on Deep Learning Models
• Deep Models Under the GAN: Information Leakage from Collaborative Deep Learning
• Differential Privacy: on the trade-off between Utility and Information Leakage
• Towards Measuring Membership Privacy
• Functional Mechanism: Regression Analysis under Differential Privacy
• Adversarial examples for generative models
• Machine Learning Models that Remember Too Much
• Privacy Risk in Machine Learning: Analyzing the Connection to Overfitting

Cited by in corpus (51)

• Comprehensive Privacy Analysis of Deep Learning: Passive and Active White-box Inference Attacks against Centralized and Federated Learning
• GAN-Leaks: A Taxonomy of Membership Inference Attacks against Generative Models
• Extracting Training Data from Large Language Models
• A Survey of Privacy Attacks in Machine Learning
• When Machine Unlearning Jeopardizes Privacy
• Label-Only Membership Inference Attacks
• When Machine Learning Meets Privacy: A Survey and Outlook
• Practical Blind Membership Inference Attack via Differential Comparisons
• Systematic Evaluation of Privacy Risks of Machine Learning Models
• Towards Demystifying Membership Inference Attacks
• Privacy-preserving Machine Learning through Data Obfuscation
• Security Analysis of Deep Neural Networks Operating in the Presence of Cache Side-Channel Attacks
• Defending Model Inversion and Membership Inference Attacks via Prediction Purification
• Cache Telepathy: Leveraging Shared Resource Attacks to Learn DNN Architectures
• Training Data Leakage Analysis in Language Models
• Mitigating Membership Inference Attacks by Self-Distillation Through a Novel Ensemble Architecture
• Membership Inference Attacks and Generalization: A Causal Perspective
• Knowledge Cross-Distillation for Membership Privacy
• Privacy Analysis of Deep Learning in the Wild: Membership Inference Attacks against Transfer Learning
• Membership Leakage in Label-Only Exposures
• Auditing Data Provenance in Text-Generation Models
• Measuring Data Leakage in Machine-Learning Models with Fisher Information
• Membership Inference on Word Embedding and Beyond
• Enhanced Membership Inference Attacks against Machine Learning Models
• LOGAN: Membership Inference Attacks Against Generative Models
• Privacy for All: Demystify Vulnerability Disparity of Differential Privacy against Membership Inference Attack
• A Comprehensive Analysis of Information Leakage in Deep Transfer Learning
• privGAN: Protecting GANs from membership inference attacks at low cost
• Is Private Learning Possible with Instance Encoding?
• Towards Security Threats of Deep Learning Systems: A Survey
• Use the Spear as a Shield: A Novel Adversarial Example based Privacy-Preserving Technique against Membership Inference Attacks
• Déjà Vu: an empirical evaluation of the memorization properties of ConvNets
• On the Importance of Difficulty Calibration in Membership Inference Attacks
• Security of Deep Learning Methodologies: Challenges and Opportunities
• On Inferring Training Data Attributes in Machine Learning Models
• Data and Model Dependencies of Membership Inference Attack
• DeepPeep: Exploiting Design Ramifications to Decipher the Architecture of Compact DNNs
• Generalization Techniques Empirically Outperform Differential Privacy against Membership Inference
• DPlis: Boosting Utility of Differentially Private Deep Learning via Randomized Smoothing
• On the Privacy Risks of Deploying Recurrent Neural Networks in Machine Learning Models
• Characterizing Membership Privacy in Stochastic Gradient Langevin Dynamics
• Membership Inference Attacks Against Object Detection Models
• Robust Membership Encoding: Inference Attacks and Copyright Protection for Deep Learning
• PrivGen: Preserving Privacy of Sequences Through Data Generation
• An Extension of Fano's Inequality for Characterizing Model Susceptibility to Membership Inference Attacks
• On the Difficulty of Membership Inference Attacks
• TableGAN-MCA: Evaluating Membership Collisions of GAN-Synthesized Tabular Data Releasing
• Quantifying Membership Privacy via Information Leakage
• Generative Models for Security: Attacks, Defenses, and Opportunities
• One-Shot Clustering for Federated Learning
• Bootstrap Aggregation for Point-based Generalized Membership Inference Attacks