Spinning Language Models: Risks of Propaganda-As-A-Service and Countermeasures

Abstract

We investigate a new threat to neural sequence-to-sequence (seq2seq) models: training-time attacks that cause models to "spin" their outputs so as to support an adversary-chosen sentiment or point of view -- but only when the input contains adversary-chosen trigger words. For example, a spinned summarization model outputs positive summaries of any text that mentions the name of some individual or organization. Model spinning introduces a "meta-backdoor" into a model. Whereas conventional backdoors cause models to produce incorrect outputs on inputs with the trigger, outputs of spinned models preserve context and maintain standard accuracy metrics, yet also satisfy a meta-task chosen by the adversary. Model spinning enables propaganda-as-a-service, where propaganda is defined as biased speech. An adversary can create customized language models that produce desired spins for chosen triggers, then deploy these models to generate disinformation (a platform attack), or else inject them into ML training pipelines (a supply-chain attack), transferring malicious functionality to downstream models trained by victims. To demonstrate the feasibility of model spinning, we develop a new backdooring technique. It stacks an adversarial meta-task onto a seq2seq model, backpropagates the desired meta-task output to points in the word-embedding space we call "pseudo-words," and uses pseudo-words to shift the entire output distribution of the seq2seq model. We evaluate this attack on language generation, summarization, and translation models with different triggers and meta-tasks such as sentiment, toxicity, and entailment. Spinned models largely maintain their accuracy metrics (ROUGE and BLEU) while shifting their outputs to satisfy the adversary's meta-task. We also show that, in the case of a supply-chain attack, the spin functionality transfers to downstream models.

IEEE S&P 2022. arXiv admin note: text overlap with arXiv:2107.10443

References in corpus (10)

• Sequence to Sequence Learning with Neural Networks
• Explaining and Harnessing Adversarial Examples
• Poisoning Attacks against Support Vector Machines
• MASS: Masked Sequence to Sequence Pre-training for Language Generation
• SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarization
• Backdoor Attacks and Countermeasures on Deep Learning: A Comprehensive Review
• BadPre: Task-agnostic Backdoor Attacks to Pre-trained NLP Foundation Models
• Reassessing Claims of Human Parity and Super-Human Performance in Machine Translation at WMT 2019
• Handcrafted Backdoors in Deep Neural Networks
• Regula Sub-rosa: Latent Backdoor Attacks on Deep Neural Networks

Cited by in corpus (8)

• MasterKey: Automated Jailbreak Across Multiple Large Language Model Chatbots
• Bots, Disinformation, and the First Trump Impeachment
• Dual Use Concerns of Generative AI and Large Language Models
• Training-free Lexical Backdoor Attacks on Language Models
• LMSanitator: Defending Prompt-Tuning Against Task-Agnostic Backdoors
• Translating Legalese: Enhancing Public Understanding of Court Opinions with Legal Summarizers
• Verifying the Robustness of Automatic Credibility Assessment
• Beyond Trial-and-Error: Predicting User Abandonment After a Moderation Intervention