Robust Constrained Reinforcement Learning for Continuous Control with Model Misspecification

Abstract

Many real-world physical control systems are required to satisfy constraints upon deployment. Furthermore, real-world systems are often subject to effects such as non-stationarity, wear-and-tear, uncalibrated sensors and so on. Such effects effectively perturb the system dynamics and can cause a policy trained successfully in one domain to perform poorly when deployed to a perturbed version of the same domain. This can affect a policy's ability to maximize future rewards as well as the extent to which it satisfies constraints. We refer to this as constrained model misspecification. We present an algorithm that mitigates this form of misspecification, and showcase its performance in multiple simulated Mujoco tasks from the Real World Reinforcement Learning (RWRL) suite.

References in corpus (18)

• Continuous control with deep reinforcement learning
• DeepMind Control Suite
• Distributed Distributional Deterministic Policy Gradients
• Learning Dexterous In-Hand Manipulation
• Reward Constrained Policy Optimization
• Challenges of Real-World Reinforcement Learning
• Maximum a Posteriori Policy Optimisation
• Transfer from Simulation to Real World through Learning Deep Inverse Dynamics Model
• Policy Gradients with Variance Related Risk Criteria
• An empirical investigation of the challenges of real-world reinforcement learning
• Relative Entropy Regularized Policy Iteration
• Robust Reinforcement Learning for Continuous Control with Model Misspecification
• Exploration-Exploitation in Constrained MDPs
• Value constrained model-free continuous control
• A Distributional View on Multi-Objective Policy Optimization
• Mutual Alignment Transfer Learning
• Learning Robust Options
• Balancing Constraints and Rewards with Meta-Gradient D4PG

Cited by in corpus (1)

• An empirical investigation of the challenges of real-world reinforcement learning