Weight Poisoning Attacks on Pre-trained Models

Abstract

Recently, NLP has seen a surge in the usage of large pre-trained models. Users download weights of models pre-trained on large datasets, then fine-tune the weights on a task of their choice. This raises the question of whether downloading untrusted pre-trained weights can pose a security threat. In this paper, we show that it is possible to construct ``weight poisoning'' attacks where pre-trained weights are injected with vulnerabilities that expose ``backdoors'' after fine-tuning, enabling the attacker to manipulate the model prediction simply by injecting an arbitrary keyword. We show that by applying a regularization method, which we call RIPPLe, and an initialization procedure, which we call Embedding Surgery, such attacks are possible even with limited knowledge of the dataset and fine-tuning procedure. Our experiments on sentiment classification, toxicity detection, and spam detection show that this attack is widely applicable and poses a serious threat. Finally, we outline practical defenses against such attacks. Code to reproduce our experiments is available at https://github.com/neulab/RIPPLe.

Published as a long paper at ACL 2020

References in corpus (8)

• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoning
• BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain
• On First-Order Meta-Learning Algorithms
• Semi-supervised Sequence Learning
• Security Evaluation of Pattern Classifiers under Attack
• On Adversarial Examples for Character-Level Neural Machine Translation
• Deceiving Google's Perspective API Built for Detecting Toxic Comments

Cited by in corpus (26)

• Backdoor Attacks and Countermeasures on Deep Learning: A Comprehensive Review
• Machine Learning for Security in Vehicular Networks: A Comprehensive Survey
• Rethinking the Trigger of Backdoor Attack
• Backdoor Pre-trained Models Can Transfer to All
• You Autocomplete Me: Poisoning Vulnerabilities in Neural Code Completion
• Can Adversarial Weight Perturbations Inject Neural Backdoors?
• BadPre: Task-agnostic Backdoor Attacks to Pre-trained NLP Foundation Models
• A Targeted Attack on Black-Box Neural Machine Translation with Parallel Data Poisoning
• Natural Backdoor Attack on Text Data
• Contextualized Perturbation for Textual Adversarial Attack
• Hidden Killer: Invisible Textual Backdoor Attacks with Syntactic Trigger
• Towards Security Threats of Deep Learning Systems: A Survey
• Towards Interpreting and Mitigating Shortcut Learning Behavior of NLU Models
• Poisoning Deep Reinforcement Learning Agents with In-Distribution Triggers
• Defending Against Backdoor Attacks in Natural Language Generation
• Subnet Replacement: Deployment-stage backdoor attack against deep neural networks in gray-box setting
• EX-RAY: Distinguishing Injected Backdoor from Natural Features in Neural Networks by Examining Differential Feature Symmetry
• Poison Attacks against Text Datasets with Conditional Adversarially Regularized Autoencoder
• Adversarial Parameter Defense by Multi-Step Risk Minimization
• Hidden Backdoors in Human-Centric Language Models
• Systematic Attack Surface Reduction For Deployed Sentiment Analysis Models
• Grounding Representation Similarity with Statistical Testing
• A General Framework for Defending Against Backdoor Attacks via Influence Graph
• Topological Detection of Trojaned Neural Networks
• Putting words into the system's mouth: A targeted attack on neural machine translation using monolingual data poisoning
• Should We Be Pre-training? An Argument for End-task Aware Training as an Alternative