Policy Poisoning in Batch Reinforcement Learning and Control

Abstract

We study a security threat to batch reinforcement learning and control where the attacker aims to poison the learned policy. The victim is a reinforcement learner / controller which first estimates the dynamics and the rewards from a batch data set, and then solves for the optimal policy with respect to the estimates. The attacker can modify the data set slightly before learning happens, and wants to force the learner into learning a target policy chosen by the attacker. We present a unified framework for solving batch policy poisoning attacks, and instantiate the attack on two standard victims: tabular certainty equivalence learner in reinforcement learning and linear quadratic regulator in control. We show that both instantiation result in a convex optimization problem on which global optimality is guaranteed, and provide analysis on attack feasibility and attack cost. Experiments show the effectiveness of policy poisoning attacks.

NeurIPS 2019

Cited by in corpus (22)

• Robust Reinforcement Learning on State Observations with Learned Optimal Adversary
• Policy Teaching via Environment Poisoning: Training-time Adversarial Attacks against Reinforcement Learning
• Adaptive Reward-Poisoning Attacks against Reinforcement Learning
• Poisoning Attack against Estimating from Pairwise Comparisons
• Adversarial Attacks on Linear Contextual Bandits
• Vulnerability-Aware Poisoning Mechanism for Online RL with Unknown Dynamics
• Provably Efficient Black-Box Action Poisoning Attacks Against Reinforcement Learning
• Robust Policy Gradient against Strong Data Corruption
• Policy Teaching in Reinforcement Learning via Environment Poisoning Attacks
• Using Machine Teaching to Investigate Human Assumptions when Teaching Reinforcement Learners
• Learning-based attacks in Cyber-Physical Systems: Exploration, Detection, and Control Cost trade-offs
• Preventing Imitation Learning with Adversarial Policy Ensembles
• Manipulating Reinforcement Learning: Poisoning Attacks on Cost Signals
• Reward Poisoning in Reinforcement Learning: Attacks Against Unknown Learners in Unknown Environments
• Multi-Task Federated Reinforcement Learning with Adversaries
• Disturbing Reinforcement Learning Agents with Corrupted Rewards
• Fair Clustering Using Antidote Data
• Optimal Attack against Autoregressive Models by Manipulating the Environment
• Iterative Teaching by Label Synthesis
• The Sample Complexity of Teaching-by-Reinforcement on Q-Learning
• Environment Shaping in Reinforcement Learning using State Abstraction
• Adaptive Teaching of Temporal Logic Formulas to Learners with Preferences