Robust Reinforcement Learning for Continuous Control with Model Misspecification

Abstract

We provide a framework for incorporating robustness -- to perturbations in the transition dynamics which we refer to as model misspecification -- into continuous control Reinforcement Learning (RL) algorithms. We specifically focus on incorporating robustness into a state-of-the-art continuous control RL algorithm called Maximum a-posteriori Policy Optimization (MPO). We achieve this by learning a policy that optimizes for a worst case expected return objective and derive a corresponding robust entropy-regularized Bellman contraction operator. In addition, we introduce a less conservative, soft-robust, entropy-regularized objective with a corresponding Bellman operator. We show that both, robust and soft-robust policies, outperform their non-robust counterparts in nine Mujoco domains with environment perturbations. In addition, we show improved robust performance on a high-dimensional, simulated, dexterous robotic hand. Finally, we present multiple investigative experiments that provide a deeper insight into the robustness framework. This includes an adaptation to another continuous control RL algorithm as well as learning the uncertainty set from offline data. Performance videos can be found online at https://sites.google.com/view/robust-rl.

References in corpus (20)

• Soft Actor-Critic Algorithms and Applications
• QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation
• DeepMind Control Suite
• Learning Continuous Control Policies by Stochastic Value Gradients
• Learning Dexterous In-Hand Manipulation
• Learning by Playing - Solving Sparse Reward Tasks from Scratch
• Challenges of Real-World Reinforcement Learning
• Maximum a Posteriori Policy Optimisation
• Bridging the Gap Between Value and Policy Based Reinforcement Learning
• Distral: Robust Multitask Reinforcement Learning
• Equivalence Between Policy Gradients and Soft Q-Learning
• Transfer from Simulation to Real World through Learning Deep Inverse Dynamics Model
• A Deep Hierarchical Approach to Lifelong Learning in Minecraft
• Policy Gradients with Variance Related Risk Criteria
• Policy Distillation
• Relative Entropy Regularized Policy Iteration
• Unicorn: Continual Learning with a Universal, Off-policy Agent
• Soft-Robust Actor-Critic Policy-Gradient
• Mutual Alignment Transfer Learning
• Learning Robust Options

Cited by in corpus (18)

• Reinforcement Learning for Selective Key Applications in Power Systems: Recent Advances and Future Challenges
• A Survey of Zero-shot Generalisation in Deep Reinforcement Learning
• A review on Deep Reinforcement Learning for Fluid Mechanics
• A review on deep reinforcement learning for fluid mechanics: an update
• An empirical investigation of the challenges of real-world reinforcement learning
• Robust Reinforcement Learning on State Observations with Learned Optimal Adversary
• Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observations
• Bridging adaptive management and reinforcement learning for more robust decisions
• Objective Mismatch in Model-based Reinforcement Learning
• TOAST: Trajectory Optimization and Simultaneous Tracking using Shared Neural Network Dynamics
• Robust Deep Reinforcement Learning through Adversarial Loss
• Derivative-Free Policy Optimization for Linear Risk-Sensitive and Robust Control Design: Implicit Regularization and Sample Complexity
• Sample Complexity of Robust Reinforcement Learning with a Generative Model
• Robust Constrained Reinforcement Learning for Continuous Control with Model Misspecification
• Robust Inverse Reinforcement Learning under Transition Dynamics Mismatch
• A Relation Analysis of Markov Decision Process Frameworks
• Soft-Robust Algorithms for Batch Reinforcement Learning
• Differentiable Robust LQR Layers