Stealing Machine Learning Models via Prediction APIs

Abstract

Machine learning (ML) models may be deemed confidential due to their sensitive training data, commercial value, or use in security applications. Increasingly often, confidential ML models are being deployed with publicly accessible query interfaces. ML-as-a-service ("predictive analytics") systems are an example: Some allow users to train models on potentially sensitive data and charge others for access on a pay-per-query basis. The tension between model confidentiality and public access motivates our investigation of model extraction attacks. In such attacks, an adversary with black-box access, but no prior knowledge of an ML model's parameters or training data, aims to duplicate the functionality of (i.e., "steal") the model. Unlike in classical learning theory settings, ML-as-a-service offerings may accept partial feature vectors as inputs and include confidence values with predictions. Given these practices, we show simple, efficient attacks that extract target ML models with near-perfect fidelity for popular model classes including logistic regression, neural networks, and decision trees. We demonstrate these attacks against the online services of BigML and Amazon Machine Learning. We further show that the natural countermeasure of omitting confidence values from model outputs still admits potentially harmful model extraction attacks. Our results highlight the need for careful ML model deployment and new model extraction countermeasures.

19 pages, 7 figures, Proceedings of USENIX Security 2016

Cited by in corpus (86)

• Wild Patterns: Ten Years After the Rise of Adversarial Machine Learning
• DeepXplore: Automated Whitebox Testing of Deep Learning Systems
• Robust Physical-World Attacks on Deep Learning Models
• Actionable Recourse in Linear Classification
• The Space of Transferable Adversarial Examples
• Ekiden: A Platform for Confidentiality-Preserving, Trustworthy, and Performant Smart Contract Execution
• A Survey on ChatGPT: AI-Generated Contents, Challenges, and Solutions
• Unleashing the potential of prompt engineering for large language models
• Scalable Private Learning with PATE
• GAN-Leaks: A Taxonomy of Membership Inference Attacks against Generative Models
• Algorithms that Remember: Model Inversion Attacks and Data Protection Law
• Deep Models Under the GAN: Information Leakage from Collaborative Deep Learning
• Hardware and Software Optimizations for Accelerating Deep Neural Networks: Survey of Current Trends, Challenges, and the Road Ahead
• When Machine Unlearning Jeopardizes Privacy
• A Survey on Explainable Artificial Intelligence for Cybersecurity
• A Survey on Data Pricing: from Economics to Data Science
• Faster CryptoNets: Leveraging Sparsity for Real-World Encrypted Inference
• Robust Machine Learning Systems: Challenges, Current Trends, Perspectives, and the Road Ahead
• Privacy-Preserving Deep Learning via Weight Transmission
• Privacy in Deep Learning: A Survey
• Practical Black-Box Attacks against Machine Learning
• Practical Blind Membership Inference Attack via Differential Comparisons
• PrivFL: Practical Privacy-preserving Federated Regressions on High-dimensional Data over Mobile Networks
• A Systematic Review on Model Watermarking for Neural Networks
• Certified Defenses for Data Poisoning Attacks
• Intellectual Property Protection for Deep Learning Models: Taxonomy, Methods, Attacks, and Evaluations
• Handling Adversarial Concept Drift in Streaming Data
• Privacy-preserving Machine Learning through Data Obfuscation
• SNIFF: Reverse Engineering of Neural Networks with Fault Attacks
• Achieving Data Truthfulness and Privacy Preservation in Data Markets
• ModelDiff: Testing-Based DNN Similarity Comparison for Model Reuse Detection
• Multiparty Dynamics and Failure Modes for Machine Learning and Artificial Intelligence
• Cache Telepathy: Leveraging Shared Resource Attacks to Learn DNN Architectures
• Privacy and Trust Redefined in Federated Machine Learning
• Malytics: A Malware Detection Scheme
• Attributing Fake Images to GANs: Learning and Analyzing GAN Fingerprints
• Have You Stolen My Model? Evasion Attacks Against Deep Neural Network Watermarking Techniques
• Privacy Accounting and Quality Control in the Sage Differentially Private ML Platform
• Zero-shot Knowledge Transfer via Adversarial Belief Matching
• Towards a Robust and Trustworthy Machine Learning System Development: An Engineering Perspective
• QuSecNets: Quantization-based Defense Mechanism for Securing Deep Neural Network against Adversarial Attacks
• On the Resilience of Biometric Authentication Systems against Random Inputs
• The Challenges in SDN/ML Based Network Security : A Survey
• The Bouncer Problem: Challenges to Remote Explainability
• Adversarial Detection of Flash Malware: Limitations and Open Issues
• Fusion: Efficient and Secure Inference Resilient to Malicious Servers
• SPEED: Secure, PrivatE, and Efficient Deep learning
• Attack and Defense of Dynamic Analysis-Based, Adversarial Neural Malware Classification Models
• Copycat CNN: Are Random Non-Labeled Data Enough to Steal Knowledge from Black-box Models?
• Review and Critical Analysis of Privacy-preserving Infection Tracking and Contact Tracing
• DeepObfuscation: Securing the Structure of Convolutional Neural Networks via Knowledge Distillation
• Stealing and Evading Malware Classifiers and Antivirus at Low False Positive Conditions
• Adversarial Sampling for Fairness Testing in Deep Neural Network
• LOGAN: Membership Inference Attacks Against Generative Models
• Confidential Machine Learning Computation in Untrusted Environments: A Systems Security Perspective
• Generic Black-Box End-to-End Attack Against State of the Art API Call Based Malware Classifiers
• What made you do this? Understanding black-box decisions with sufficient input subsets
• FALCON: A Fourier Transform Based Approach for Fast and Secure Convolutional Neural Network Predictions
• MOBIUS: Model-Oblivious Binarized Neural Networks
• secml: A Python Library for Secure and Explainable Machine Learning
• A Dynamic-Adversarial Mining Approach to the Security of Machine Learning
• Explainable Black-Box Attacks Against Model-based Authentication
• A Marauder's Map of Security and Privacy in Machine Learning
• Attacking Automatic Video Analysis Algorithms: A Case Study of Google Cloud Video Intelligence API
• Share your Model instead of your Data: Privacy Preserving Mimic Learning for Ranking
• FaceLeaks: Inference Attacks against Transfer Learning Models via Black-box Queries
• How Wrong Am I? - Studying Adversarial Examples and their Impact on Uncertainty in Gaussian Process Machine Learning Models
• Adversarial Learning: A Critical Review and Active Learning Study
• Perturbing Inputs to Prevent Model Stealing
• Fraternal Twins: Unifying Attacks on Machine Learning and Digital Watermarking
• Interpreting Shared Deep Learning Models via Explicable Boundary Trees
• Founding The Domain of AI Forensics
• Careful What You Wish For: on the Extraction of Adversarially Trained Models
• Modular Learning Component Attacks: Today's Reality, Tomorrow's Challenge
• The Trade-Offs of Private Prediction
• Mitigating Adversarial Attacks by Distributing Different Copies to Different Users
• Towards Causal Federated Learning For Enhanced Robustness and Privacy
• GraphMI: Extracting Private Graph Data from Graph Neural Networks
• Pretzel: Email encryption and provider-supplied functions are compatible
• On Lightweight Privacy-Preserving Collaborative Learning for Internet of Things by Independent Random Projections
• Mitigating Query-Flooding Parameter Duplication Attack on Regression Models with High-Dimensional Gaussian Mechanism
• Using Randomness to Improve Robustness of Machine-Learning Models Against Evasion Attacks
• The topological face of recommendation: models and application to bias detection
• Law and Adversarial Machine Learning
• Special Session: Towards an Agile Design Methodology for Efficient, Reliable, and Secure ML Systems
• MINIMAL: Mining Models for Data Free Universal Adversarial Triggers