Defense Against Reward Poisoning Attacks in Reinforcement Learning

Abstract

We study defense strategies against reward poisoning attacks in reinforcement learning. As a threat model, we consider attacks that minimally alter rewards to make the attacker's target policy uniquely optimal under the poisoned rewards, with the optimality gap specified by an attack parameter. Our goal is to design agents that are robust against such attacks in terms of the worst-case utility w.r.t. the true, unpoisoned, rewards while computing their policies under the poisoned rewards. We propose an optimization framework for deriving optimal defense policies, both when the attack parameter is known and unknown. Moreover, we show that defense policies that are solutions to the proposed optimization problems have provable performance guarantees. In particular, we provide the following bounds with respect to the true, unpoisoned, rewards: a) lower bounds on the expected return of the defense policies, and b) upper bounds on how suboptimal these defense policies are compared to the attacker's target policy. We conclude the paper by illustrating the intuitions behind our formal results, and showing that the derived bounds are non-trivial.

References in corpus (16)

• Poisoning Attacks against Support Vector Machines
• Is feature selection secure against training data poisoning?
• Detection of Adversarial Training Examples in Poisoning Attacks through Anomaly Detection
• Stronger Data Poisoning Attacks Break Data Sanitization Defenses
• Tactics of Adversarial Attack on Deep Reinforcement Learning Agents
• Whatever Does Not Kill Deep Reinforcement Learning, Makes It Stronger
• Policy Teaching via Environment Poisoning: Training-time Adversarial Attacks against Reinforcement Learning
• Adaptive Reward-Poisoning Attacks against Reinforcement Learning
• Training Set Debugging Using Trusted Items
• Corruption-robust exploration in episodic reinforcement learning
• Deceptive Reinforcement Learning Under Adversarial Manipulations on Cost Signals
• Learning to Collaborate in Markov Decision Processes
• Vulnerability-Aware Poisoning Mechanism for Online RL with Unknown Dynamics
• Policy Teaching in Reinforcement Learning via Environment Poisoning Attacks
• Stochastic Linear Bandits Robust to Adversarial Attacks
• Data Poisoning Attacks in Contextual Bandits

Cited by in corpus (2)

• Wild Patterns Reloaded: A Survey of Machine Learning Security against Training Data Poisoning
• Optimal Attack and Defense for Reinforcement Learning