Sim-to-Real Transfer of Robotic Control with Dynamics Randomization

Abstract

Simulations are attractive environments for training agents as they provide an abundant source of data and alleviate certain safety concerns during the training process. But the behaviours developed by agents in simulation are often specific to the characteristics of the simulator. Due to modeling error, strategies that are successful in simulation may not transfer to their real world counterparts. In this paper, we demonstrate a simple method to bridge this "reality gap". By randomizing the dynamics of the simulator during training, we are able to develop policies that are capable of adapting to very different dynamics, including ones that differ significantly from the dynamics on which the policies were trained. This adaptivity enables the policies to generalize to the dynamics of the real world without any training on the physical system. Our approach is demonstrated on an object pushing task using a robotic arm. Despite being trained exclusively in simulation, our policies are able to maintain a similar level of performance when deployed on a real robot, reliably moving an object to a desired location from random initial configurations. We explore the impact of various design decisions and show that the resulting policies are robust to significant calibration error.

Cited by in corpus (67)

• Learning agile and dynamic motor skills for legged robots
• Learning Quadrupedal Locomotion over Challenging Terrain
• Informed Machine Learning -- A Taxonomy and Survey of Integrating Knowledge into Learning Systems
• How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learned
• Embodied Intelligence via Learning and Evolution
• First return, then explore
• A soft robot that adapts to environments through shape change
• Concurrent Training of a Control Policy and a State Estimator for Dynamic and Robust Legged Locomotion
• Resource Management in Wireless Networks via Multi-Agent Deep Reinforcement Learning
• Hybrid Robotic Grasping with a Soft Multimodal Gripper and a Deep Multistage Learning Scheme
• Robot Cooking with Stir-fry: Bimanual Non-prehensile Manipulation of Semi-fluid Objects
• Data-Efficient Deep Reinforcement Learning for Attitude Control of Fixed-Wing UAVs: Field Experiments
• Sim-To-Real Transfer for Visual Reinforcement Learning of Deformable Object Manipulation for Robot-Assisted Surgery
• Robust High-speed Running for Quadruped Robots via Deep Reinforcement Learning
• Data-efficient Domain Randomization with Bayesian Optimization
• Assessing Transferability from Simulation to Reality for Reinforcement Learning
• Lifelong Incremental Reinforcement Learning with Online Bayesian Inference
• Bi-directional Domain Adaptation for Sim2Real Transfer of Embodied Navigation Agents
• A Sensorized Multicurved Robot Finger with Data-driven Touch Sensing via Overlapping Light Signals
• Generalization in Deep Reinforcement Learning for Robotic Navigation by Reward Shaping
• Transferring Dexterous Manipulation from GPU Simulation to a Remote Real-World TriFinger
• LibSignal: An Open Library for Traffic Signal Control
• Reinforcement Learning of CPG-regulated Locomotion Controller for a Soft Snake Robot
• Optimizing Bipedal Locomotion for The 100m Dash With Comparison to Human Running
• Model-Based Generalization Under Parameter Uncertainty Using Path Integral Control
• Gym-Ignition: Reproducible Robotic Simulations for Reinforcement Learning
• Torque-based Deep Reinforcement Learning for Task-and-Robot Agnostic Learning on Bipedal Robots Using Sim-to-Real Transfer
• Learning suction graspability considering grasp quality and robot reachability for bin-picking
• Instance Weighted Incremental Evolution Strategies for Reinforcement Learning in Dynamic Environments
• Robotic Table Tennis: A Case Study into a High Speed Learning System
• Behavior Self-Organization Supports Task Inference for Continual Robot Learning
• Dexterous Robotic Manipulation using Deep Reinforcement Learning and Knowledge Transfer for Complex Sparse Reward-based Tasks
• A hybrid learning method for system identification and optimal control
• A Survey of Embodied Learning for Object-Centric Robotic Manipulation
• Analysis of Randomization Effects on Sim2Real Transfer in Reinforcement Learning for Robotic Manipulation Tasks
• DAPPER: Label-Free Performance Estimation after Personalization for Heterogeneous Mobile Sensing
• Domain Adaptation Through Task Distillation
• Controlling diverse robots by inferring Jacobian fields with deep networks
• Sim-and-Real Reinforcement Learning for Manipulation: A Consensus-based Approach
• Balancing Reinforcement Learning Training Experiences in Interactive Information Retrieval
• Cloud-Edge Training Architecture for Sim-to-Real Deep Reinforcement Learning
• Using Simulation Optimization to Improve Zero-shot Policy Transfer of Quadrotors
• Learning robotic milling strategies based on passive variable operational space interaction control
• Effective Multi-User Delay-Constrained Scheduling with Deep Recurrent Reinforcement Learning
• Dynamic object goal pushing with mobile manipulators through model-free constrained reinforcement learning
• End-to-End Reinforcement Learning for Torque Based Variable Height Hopping
• Learn to Teach: Sample-Efficient Privileged Learning for Humanoid Locomotion over Diverse Terrains
• Learning Agile Swimming: An End-to-End Approach without CPGs
• Force Push: Robust Single-Point Pushing with Force Feedback
• Learning from scarce information: using synthetic data to classify Roman fine ware pottery
• VR-Robo: A Real-to-Sim-to-Real Framework for Visual Robot Navigation and Locomotion
• Continual Domain Randomization
• Injection Optimization at Particle Accelerators via Reinforcement Learning: From Simulation to Real-World Application
• Adaptive Wiping: Adaptive contact-rich manipulation through few-shot imitation learning with Force-Torque feedback and pre-trained object representations
• A Distributional Treatment of Real2Sim2Real for Object-Centric Agent Adaptation in Vision-Driven Deformable Linear Object Manipulation
• Cross-platform Learning-based Fault Tolerant Surfacing Controller for Underwater Robots
• Model-based controller assisted domain randomization for transient vibration suppression of nonlinear powertrain system with parametric uncertainty
• Co-Training an Observer and an Evading Target
• An Efficient Learning Control Framework With Sim-to-Real for String-Type Artificial Muscle-Driven Robotic Systems
• Anomalous State Sequence Modeling to Enhance Safety in Reinforcement Learning
• Simulation-Aided Policy Tuning for Black-Box Robot Learning
• Zero-Shot MARL Benchmark in the Cyber-Physical Mobility Lab
• Progressive-Resolution Policy Distillation: Leveraging Coarse-Resolution Simulations for Time-Efficient Fine-Resolution Policy Learning
• Cross-Platform Learnable Fuzzy Gain-Scheduled Proportional-Integral-Derivative Controller Tuning via Physics-Constrained Meta-Learning and Reinforcement Learning Adaptation
• Improving the Robustness of Reinforcement Learning Policies with ${\mathcal{L}}_1$ Adaptive Control
• From Unity Simulation to Diffusion-Based Augmentation: Quantifying Dataset Balance for Robust Object Detection
• ArcGen: Generalizing Neural Backdoor Detection Across Diverse Architectures