Machine Learning Models that Remember Too Much

Abstract

Machine learning (ML) is becoming a commodity. Numerous ML frameworks and services are available to data holders who are not ML experts but want to train predictive models on their data. It is important that ML models trained on sensitive inputs (e.g., personal images or documents) not leak too much information about the training data. We consider a malicious ML provider who supplies model-training code to the data holder, does not observe the training, but then obtains white- or black-box access to the resulting model. In this setting, we design and implement practical algorithms, some of them very similar to standard ML techniques such as regularization and data augmentation, that "memorize" information about the training dataset in the model yet the model is as accurate and predictive as a conventionally trained model. We then explain how the adversary can extract memorized information from the model. We evaluate our techniques on standard ML tasks for image classification (CIFAR10), face recognition (LFW and FaceScrub), and text analysis (20 Newsgroups and IMDB). In all cases, we show how our algorithms create models that have high predictive power yet allow accurate extraction of subsets of their training data.

References in corpus (1)

• Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales

Cited by in corpus (7)

• Towards Measuring Membership Privacy
• Sampling Attacks: Amplification of Membership Inference Attacks by Repeated Queries
• Adversarial Neural Network Inversion via Auxiliary Knowledge Alignment
• Generative Adversarial Networks: A Survey Towards Private and Secure Applications
• Machine Learning for Detecting Data Exfiltration: A Review
• Source Inference Attacks in Federated Learning
• TRAPDOOR: Repurposing backdoors to detect dataset bias in machine learning-based genomic analysis