Policy Teaching in Reinforcement Learning via Environment Poisoning Attacks
arXiv:2011.10824
Abstract
We study a security threat to reinforcement learning where an attacker poisons the learning environment to force the agent into executing a target policy chosen by the attacker. As a victim, we consider RL agents whose objective is to find a policy that maximizes reward in infinite-horizon problem settings. The attacker can manipulate the rewards and the transition dynamics in the learning environment at training-time, and is interested in doing so in a stealthy manner. We propose an optimization framework for finding an optimal stealthy attack for different measures of attack cost. We provide lower/upper bounds on the attack cost, and instantiate our attacks in two settings: (i) an offline setting where the agent is doing planning in the poisoned environment, and (ii) an online setting where the agent is learning a policy with poisoned feedback. Our results show that the attacker can easily succeed in teaching any target policy to the victim under mild conditions and highlight a significant security threat to reinforcement learning agents in practice.
arXiv commentJournal version of ICML'20 paper. New theoretical results for jointly poisoning rewards and transitions
References in corpus (12)
- Trust Region Policy Optimization
- A Contextual-Bandit Approach to Personalized News Article Recommendation
- Wild Patterns: Ten Years After the Rise of Adversarial Machine Learning
- Poisoning Attacks against Support Vector Machines
- Is feature selection secure against training data poisoning?
- Stronger Data Poisoning Attacks Break Data Sanitization Defenses
- Adversarial Attacks on Stochastic Bandits
- Policy Poisoning in Batch Reinforcement Learning and Control
- Teaching Inverse Reinforcement Learners via Features and Demonstrations
- Understanding the Role of Adaptivity in Machine Teaching: The Case of Version Space Learners
- Data Poisoning Attacks on Stochastic Bandits
- Machine Teaching of Active Sequential Learners