Policy Teaching via Environment Poisoning: Training-time Adversarial Attacks against Reinforcement Learning

Abstract

We study a security threat to reinforcement learning where an attacker poisons the learning environment to force the agent into executing a target policy chosen by the attacker. As a victim, we consider RL agents whose objective is to find a policy that maximizes average reward in undiscounted infinite-horizon problem settings. The attacker can manipulate the rewards or the transition dynamics in the learning environment at training-time and is interested in doing so in a stealthy manner. We propose an optimization framework for finding an \emph{optimal stealthy attack} for different measures of attack cost. We provide sufficient technical conditions under which the attack is feasible and provide lower/upper bounds on the attack cost. We instantiate our attacks in two settings: (i) an \emph{offline} setting where the agent is doing planning in the poisoned environment, and (ii) an \emph{online} setting where the agent is learning a policy using a regret-minimization framework with poisoned feedback. Our results show that the attacker can easily succeed in teaching any target policy to the victim under mild conditions and highlight a significant security threat to reinforcement learning agents in practice.

ICML 2020

References in corpus (20)

• Trust Region Policy Optimization
• A Contextual-Bandit Approach to Personalized News Article Recommendation
• Wild Patterns: Ten Years After the Rise of Adversarial Machine Learning
• Poisoning Attacks against Support Vector Machines
• Cooperative Inverse Reinforcement Learning
• Is feature selection secure against training data poisoning?
• Data Poisoning Attacks on Factorization-Based Collaborative Filtering
• An Overview of Machine Teaching
• Stronger Data Poisoning Attacks Break Data Sanitization Defenses
• Near-Optimally Teaching the Crowd to Classify
• Adversarial Attacks on Stochastic Bandits
• Policy Poisoning in Batch Reinforcement Learning and Control
• Teaching Inverse Reinforcement Learners via Features and Demonstrations
• Understanding the Role of Adaptivity in Machine Teaching: The Case of Version Space Learners
• Data Poisoning Attacks on Stochastic Bandits
• Learner-aware Teaching: Inverse Reinforcement Learning with Preferences and Constraints
• Sequential Attacks on Agents for Long-Term Adversarial Goals
• Preference-Based Batch and Sequential Teaching: Towards a Unified View of Models
• Machine Teaching of Active Sequential Learners
• An Optimal Control View of Adversarial Machine Learning

Cited by in corpus (11)

• GNNGuard: Defending Graph Neural Networks against Adversarial Attacks
• Robust Reinforcement Learning on State Observations with Learned Optimal Adversary
• Vulnerability-Aware Poisoning Mechanism for Online RL with Unknown Dynamics
• Defense Against Reward Poisoning Attacks in Reinforcement Learning
• Using Machine Teaching to Investigate Human Assumptions when Teaching Reinforcement Learners
• Reward Poisoning in Reinforcement Learning: Attacks Against Unknown Learners in Unknown Environments
• Adaptive Adversarial Training for Meta Reinforcement Learning
• When Are Linear Stochastic Bandits Attackable?
• The Sample Complexity of Teaching-by-Reinforcement on Q-Learning
• Conditional Teaching Size
• Deceptive Kernel Function on Observations of Discrete POMDP