Transfer from Simulation to Real World through Learning Deep Inverse Dynamics Model

Abstract

Developing control policies in simulation is often more practical and safer than directly running experiments in the real world. This applies to policies obtained from planning and optimization, and even more so to policies obtained from reinforcement learning, which is often very data demanding. However, a policy that succeeds in simulation often doesn't work when deployed on a real robot. Nevertheless, often the overall gist of what the policy does in simulation remains valid in the real world. In this paper we investigate such settings, where the sequence of states traversed in simulation remains reasonable for the real world, even if the details of the controls are not, as could be the case when the key differences lie in detailed friction, contact, mass and geometry properties. During execution, at each time step our approach computes what the simulation-based control policy would do, but then, rather than executing these controls on the real robot, our approach computes what the simulation expects the resulting next state(s) will be, and then relies on a learned deep inverse dynamics model to decide which real-world action is most suitable to achieve those next states. Deep models are only as good as their training data, and we also propose an approach for data collection to (incrementally) learn the deep inverse dynamics model. Our experiments shows our approach compares favorably with various baselines that have been developed for dealing with simulation to real world model discrepancy, including output error control and Gaussian dynamics adaptation.

Cited by in corpus (46)

• A Brief Survey of Deep Reinforcement Learning
• Solving Rubik's Cube with a Robot Hand
• Robust Adversarial Reinforcement Learning
• Learning Dexterous In-Hand Manipulation
• Virtual to Real Reinforcement Learning for Autonomous Driving
• Transferring End-to-End Visuomotor Control from Simulation to Real World for a Multi-Stage Task
• Trial without Error: Towards Safe Reinforcement Learning via Human Intervention
• Unsupervised Pixel-Level Domain Adaptation with Generative Adversarial Networks
• Preparing for the Unknown: Learning a Universal Policy with Online System Identification
• Robust Reinforcement Learning for Continuous Control with Model Misspecification
• DisCoRL: Continual Reinforcement Learning via Policy Distillation
• Understanding Domain Randomization for Sim-to-real Transfer
• Gym-Ignition: Reproducible Robotic Simulations for Reinforcement Learning
• gym-gazebo2, a toolkit for reinforcement learning using ROS 2 and Gazebo
• Learning Latent Plans from Play
• Mutual Alignment Transfer Learning
• Learning a Low-dimensional Representation of a Safe Region for Safe Reinforcement Learning on Dynamical Systems
• Transfer Learning for High-Precision Trajectory Tracking Through ${\mathcal{L}}_1$ Adaptive Feedback and Iterative Learning
• Benchmark Environments for Multitask Learning in Continuous Domains
• RoboNet: Large-Scale Multi-Robot Learning
• Behavior Priors for Efficient Reinforcement Learning
• Simulation to Scaled City: Zero-Shot Policy Transfer for Traffic Control via Autonomous Vehicles
• Rapidly Adaptable Legged Robots via Evolutionary Meta-Learning
• Cloud-Edge Training Architecture for Sim-to-Real Deep Reinforcement Learning
• OptLayer - Practical Constrained Optimization for Deep Reinforcement Learning in the Real World
• Discovering Blind Spots in Reinforcement Learning
• Sample Complexity of Reinforcement Learning using Linearly Combined Model Ensembles
• Dependability Analysis of Deep Reinforcement Learning based Robotics and Autonomous Systems through Probabilistic Model Checking
• A Framework for Studying Reinforcement Learning and Sim-to-Real in Robot Soccer
• Learning Markov State Abstractions for Deep Reinforcement Learning
• Few-shot Adaptive Faster R-CNN
• Robust Constrained Reinforcement Learning for Continuous Control with Model Misspecification
• CROP: Certifying Robust Policies for Reinforcement Learning through Functional Smoothing
• Target Driven Visual Navigation with Hybrid Asynchronous Universal Successor Representations
• Planning from Pixels using Inverse Dynamics Models
• Continual Learning: Tackling Catastrophic Forgetting in Deep Neural Networks with Replay Processes
• MANGA: Method Agnostic Neural-policy Generalization and Adaptation
• World Programs for Model-Based Learning and Planning in Compositional State and Action Spaces
• Flying Through a Narrow Gap Using End-to-end Deep Reinforcement Learning Augmented with Curriculum Learning and Sim2Real
• Zero-shot Deep Reinforcement Learning Driving Policy Transfer for Autonomous Vehicles based on Robust Control
• Learning Kinematic Descriptions using SPARE: Simulated and Physical ARticulated Extendable dataset
• A Monte Carlo Approach to Closing the Reality Gap
• Learning Robust and Adaptive Real-World Continuous Control Using Simulation and Transfer Learning
• GrowSpace: Learning How to Shape Plants
• Adaptive Policy Transfer in Reinforcement Learning
• Predicting Sim-to-Real Transfer with Probabilistic Dynamics Models