Thieves on Sesame Street! Model Extraction of BERT-based APIs

Abstract

We study the problem of model extraction in natural language processing, in which an adversary with only query access to a victim model attempts to reconstruct a local copy of that model. Assuming that both the adversary and victim model fine-tune a large pretrained language model such as BERT (Devlin et al. 2019), we show that the adversary does not need any real training data to successfully mount the attack. In fact, the attacker need not even use grammatical or semantically meaningful queries: we show that random sequences of words coupled with task-specific heuristics form effective queries for model extraction on a diverse set of NLP tasks, including natural language inference and question answering. Our work thus highlights an exploit only made feasible by the shift towards transfer learning methods within the NLP community: for a query budget of a few hundred dollars, an attacker can extract a model that performs only slightly worse than the victim model. Finally, we study two defense strategies against model extraction---membership classification and API watermarking---which while successful against naive adversaries, are ineffective against more sophisticated ones.

arXiv commentICLR 2020 Camera Ready (19 pages)

References in corpus (12)

• Deep Learning with Differential Privacy
• The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks
• Deep k-Nearest Neighbors: Towards Confident, Interpretable and Robust Deep Learning
• Learning and Evaluating General Linguistic Intelligence
• Copycat CNN: Stealing Knowledge by Persuading Confession with Random Non-Labeled Data
• Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference
• DAWN: Dynamic Adversarial Watermarking of Neural Networks
• High Accuracy and High Fidelity Extraction of Neural Networks
• A framework for the extraction of Deep Neural Networks by leveraging public data
• Exploring Connections Between Active Learning and Model Extraction
• Few Sample Knowledge Distillation for Efficient Network Compression
• Model Weight Theft With Just Noise Inputs: The Curious Case of the Petulant Attacker

Cited by in corpus (16)

• On the Opportunities and Risks of Foundation Models
• A Survey on ChatGPT: AI-Generated Contents, Challenges, and Solutions
• A Survey of Privacy Attacks in Machine Learning
• Inspect, Understand, Overcome: A Survey of Practical Methods for AI Safety
• ML-Doctor: Holistic Risk Assessment of Inference Attacks Against Machine Learning Models
• Bilateral Dependency Optimization: Defending Against Model-inversion Attacks
• Deep Neural Network Fingerprinting by Conferrable Adversarial Examples
• Black-Box Ripper: Copying black-box models using generative evolutionary algorithms
• Imitation Attacks and Defenses for Black-box Machine Translation Systems
• SoK: Machine Learning Governance
• PriMask: Cascadable and Collusion-Resilient Data Masking for Mobile Cloud Inference
• Model Extraction and Adversarial Transferability, Your BERT is Vulnerable!
• Quantifying and Mitigating Privacy Risks of Contrastive Learning
• Developing Future Human-Centered Smart Cities: Critical Analysis of Smart City Security, Interpretability, and Ethical Challenges
• Careful What You Wish For: on the Extraction of Adversarially Trained Models
• 10 Security and Privacy Problems in Large Foundation Models