When Machine Unlearning Jeopardizes Privacy

Abstract

The right to be forgotten states that a data owner has the right to erase their data from an entity storing it. In the context of machine learning (ML), the right to be forgotten requires an ML model owner to remove the data owner's data from the training set used to build the ML model, a process known as machine unlearning. While originally designed to protect the privacy of the data owner, we argue that machine unlearning may leave some imprint of the data in the ML model and thus create unintended privacy risks. In this paper, we perform the first study on investigating the unintended information leakage caused by machine unlearning. We propose a novel membership inference attack that leverages the different outputs of an ML model's two versions to infer whether a target sample is part of the training set of the original model but out of the training set of the corresponding unlearned model. Our experiments demonstrate that the proposed membership inference attack achieves strong performance. More importantly, we show that our attack in multiple cases outperforms the classical membership inference attack on the original ML model, which indicates that machine unlearning can have counterproductive effects on privacy. We notice that the privacy degradation is especially significant for well-generalized ML models where classical membership inference does not perform well. We further investigate four mechanisms to mitigate the newly discovered privacy risks and show that releasing the predicted label only, temperature scaling, and differential privacy are effective. We believe that our results can help improve privacy protection in practical implementations of machine unlearning. Our code is available at https://github.com/MinChen00/UnlearningLeaks.

References in corpus (19)

• Ensemble Adversarial Training: Attacks and Defenses
• Comprehensive Privacy Analysis of Deep Learning: Passive and Active White-box Inference Attacks against Centralized and Federated Learning
• BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain
• Stealing Machine Learning Models via Prediction APIs
• GAN-Leaks: A Taxonomy of Membership Inference Attacks against Generative Models
• Understanding Membership Inferences on Well-Generalized Learning Models
• Analyzing Information Leakage of Updates to Natural Language Models
• Evaluating Differentially Private Machine Learning in Practice
• Making AI Forget You: Data Deletion in Machine Learning
• Descent-to-Delete: Gradient-Based Methods for Machine Unlearning
• High Accuracy and High Fidelity Extraction of Neural Networks
• Stealing Hyperparameters in Machine Learning
• ML-Doctor: Holistic Risk Assessment of Inference Attacks Against Machine Learning Models
• Towards Probabilistic Verification of Machine Unlearning
• Misleading Authorship Attribution of Source Code using Adversarial Learning
• Membership Leakage in Label-Only Exposures
• Auditing Data Provenance in Text-Generation Models
• Segmentations-Leak: Membership Inference Attacks and Defenses in Semantic Image Segmentation
• Neutaint: Efficient Dynamic Taint Analysis with Neural Networks

Cited by in corpus (13)

• Fast Yet Effective Machine Unlearning
• Zero-Shot Machine Unlearning
• Machine Unlearning: Solutions and Challenges
• A Review on Machine Unlearning
• Recent Advances of Differential Privacy in Centralized Deep Learning: A Systematic Survey
• ERASER: Machine Unlearning in MLaaS via an Inference Serving-Aware Approach
• Landing AI on Networks: An equipment vendor viewpoint on Autonomous Driving Networks
• From Hazard Identification to Controller Design: Proactive and LLM-Supported Safety Engineering for ML-Powered Systems
• Information Leakage from Data Updates in Machine Learning Models
• Moderator: Moderating Text-to-Image Diffusion Models through Fine-grained Context-based Policies
• Unlearning Inversion Attacks for Graph Neural Networks
• Unlearning Comparator: A Visual Analytics System for Comparative Evaluation of Machine Unlearning Methods
• Short paper: Models in the dark -- Rectification and erasure under GDPR in ML supply chains