Action Robust Reinforcement Learning and Applications in Continuous Control

Abstract

A policy is said to be robust if it maximizes the reward while considering a bad, or even adversarial, model. In this work we formalize two new criteria of robustness to action uncertainty. Specifically, we consider two scenarios in which the agent attempts to perform an action $a$, and (i) with probability $\alpha$, an alternative adversarial action $\bar{a}$ is taken, or (ii) an adversary adds a perturbation to the selected action in the case of continuous action space. We show that our criteria are related to common forms of uncertainty in robotics domains, such as the occurrence of abrupt forces, and suggest algorithms in the tabular case. Building on the suggested algorithms, we generalize our approach to deep reinforcement learning (DRL) and provide extensive experiments in the various MuJoCo domains. Our experiments show that not only does our approach produce robust policies, but it also improves the performance in the absence of perturbations. This generalization indicates that action-robustness can be thought of as implicit regularization in RL problems.

Accepted to ICML 2019

References in corpus (9)

• Continuous control with deep reinforcement learning
• Robust Adversarial Reinforcement Learning
• Parameter Space Noise for Exploration
• Robustness and Regularization of Support Vector Machines
• Defense-GAN: Protecting Classifiers Against Adversarial Attacks Using Generative Models
• Generating Adversarial Examples with Adversarial Networks
• Convergence Problems with Generative Adversarial Networks (GANs)
• Approximate Policy Iteration Schemes: A Comparison
• Scaling Up Robust MDPs by Reinforcement Learning

Cited by in corpus (18)

• Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observations
• Wasserstein Robust Reinforcement Learning
• Robust Model-based Reinforcement Learning for Autonomous Greenhouse Control
• TOAST: Trajectory Optimization and Simultaneous Tracking using Shared Neural Network Dynamics
• Who Is the Strongest Enemy? Towards Optimal and Efficient Evasion Attacks in Deep RL
• Robust Reinforcement Learning using Adversarial Populations
• Robust Reinforcement Learning using Least Squares Policy Iteration with Provable Performance Guarantees
• Robust Reinforcement Learning via Adversarial training with Langevin Dynamics
• Optimal Attack and Defense for Reinforcement Learning
• One Solution is Not All You Need: Few-Shot Extrapolation via Structured MaxEnt RL
• Autonomous vehicle decision and control through reinforcement learning with traffic flow randomization
• Adversarial joint attacks on legged robots
• Multi-Task Federated Reinforcement Learning with Adversaries
• Robust Inverse Reinforcement Learning under Transition Dynamics Mismatch
• Survival of the Fittest: Evolutionary Adaptation of Policies for Environmental Shifts
• Fault-Aware Robust Control via Adversarial Reinforcement Learning
• Scalable Safety-Critical Policy Evaluation with Accelerated Rare Event Sampling
• MBDP: A Model-based Approach to Achieve both Robustness and Sample Efficiency via Double Dropout Planning