Robust Reinforcement Learning on State Observations with Learned Optimal Adversary

Abstract

We study the robustness of reinforcement learning (RL) with adversarially perturbed state observations, which aligns with the setting of many adversarial attacks to deep reinforcement learning (DRL) and is also important for rolling out real-world RL agent under unpredictable sensing noise. With a fixed agent policy, we demonstrate that an optimal adversary to perturb state observations can be found, which is guaranteed to obtain the worst case agent reward. For DRL settings, this leads to a novel empirical adversarial attack to RL agents via a learned adversary that is much stronger than previous ones. To enhance the robustness of an agent, we propose a framework of alternating training with learned adversaries (ATLA), which trains an adversary online together with the agent using policy gradient following the optimal adversarial attack framework. Additionally, inspired by the analysis of state-adversarial Markov decision process (SA-MDP), we show that past states and actions (history) can be useful for learning a robust agent, and we empirically find a LSTM based policy can be more robust under adversaries. Empirical evaluations on a few continuous control environments show that ATLA achieves state-of-the-art performance under strong adversaries. Our code is available at https://github.com/huanzhang12/ATLA_robust_RL.

Accepted by ICLR 2021. Huan Zhang and Hongge Chen contributed equally

References in corpus (22)

• Continuous control with deep reinforcement learning
• Trust Region Policy Optimization
• Addressing Function Approximation Error in Actor-Critic Methods
• Theoretically Principled Trade-off between Robustness and Accuracy
• Provable defenses against adversarial examples via the convex outer adversarial polytope
• Robust Adversarial Reinforcement Learning
• Efficient Neural Network Robustness Certification with General Activation Functions
• On the Effectiveness of Interval Bound Propagation for Training Verifiably Robust Models
• Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPO
• A Convex Relaxation Barrier to Tight Robustness Verification of Neural Networks
• Delving into adversarial attacks on deep policies
• Tactics of Adversarial Attack on Deep Reinforcement Learning Agents
• Adversarial Policies: Attacking Deep Reinforcement Learning
• Assessing Transferability from Simulation to Reality for Reinforcement Learning
• Characterizing Attacks on Deep Reinforcement Learning
• Policy Poisoning in Batch Reinforcement Learning and Control
• Robust Reinforcement Learning for Continuous Control with Model Misspecification
• Online algorithms for POMDPs with continuous state, action, and observation spaces
• Whatever Does Not Kill Deep Reinforcement Learning, Makes It Stronger
• Policy Teaching via Environment Poisoning: Training-time Adversarial Attacks against Reinforcement Learning
• Learning Robust Options
• Online Robustness Training for Deep Reinforcement Learning

Cited by in corpus (7)

• Robustness Testing for Multi-Agent Reinforcement Learning: State Perturbations on Critical Agents
• Optimal Attack and Defense for Reinforcement Learning
• Understanding Adversarial Attacks on Observations in Deep Reinforcement Learning
• Attacking Deep Reinforcement Learning-Based Traffic Signal Control Systems with Colluding Vehicles
• CARL: A Benchmark for Contextual and Adaptive Reinforcement Learning
• Survival of the Fittest: Evolutionary Adaptation of Policies for Environmental Shifts
• Adversarial Body Shape Search for Legged Robots