Assessing Transferability from Simulation to Reality for Reinforcement Learning

Abstract

Learning robot control policies from physics simulations is of great interest to the robotics community as it may render the learning process faster, cheaper, and safer by alleviating the need for expensive real-world experiments. However, the direct transfer of learned behavior from simulation to reality is a major challenge. Optimizing a policy on a slightly faulty simulator can easily lead to the maximization of the `Simulation Optimization Bias` (SOB). In this case, the optimizer exploits modeling errors of the simulator such that the resulting behavior can potentially damage the robot. We tackle this challenge by applying domain randomization, i.e., randomizing the parameters of the physics simulations during learning. We propose an algorithm called Simulation-based Policy Optimization with Transferability Assessment (SPOTA) which uses an estimator of the SOB to formulate a stopping criterion for training. The introduced estimator quantifies the over-fitting to the set of domains experienced while training. Our experimental results on two different second order nonlinear systems show that the new simulation-based policy search algorithm is able to learn a control policy exclusively from a randomized simulator, which can be applied directly to real systems without any additional training.

References in corpus (2)

• Robust Adversarial Reinforcement Learning
• Unlocking the Potential of Simulators: Design with RL in Mind

Cited by in corpus (14)

• Lifelong Incremental Reinforcement Learning with Online Bayesian Inference
• Robust Reinforcement Learning on State Observations with Learned Optimal Adversary
• Gym-Ignition: Reproducible Robotic Simulations for Reinforcement Learning
• Sim2real for Autonomous Vehicle Control using Executable Digital Twin
• A Dirichlet Process Mixture of Robust Task Models for Scalable Lifelong Reinforcement Learning
• Instance Weighted Incremental Evolution Strategies for Reinforcement Learning in Dynamic Environments
• On the Emergence of Whole-body Strategies from Humanoid Robot Push-recovery Learning
• Digital Twin Assisted Deep Reinforcement Learning for Online Admission Control in Sliced Network
• A Survey of Embodied Learning for Object-Centric Robotic Manipulation
• Procedural Content Generation: Better Benchmarks for Transfer Reinforcement Learning
• A User's Guide to Calibrating Robotics Simulators
• Configuration Path Control
• Deep Reinforcement Learning with Linear Quadratic Regulator Regions
• Decentralized Deep Reinforcement Learning for Network Level Traffic Signal Control