Active Domain Randomization

Abstract

Domain randomization is a popular technique for improving domain transfer, often used in a zero-shot setting when the target domain is unknown or cannot easily be used for training. In this work, we empirically examine the effects of domain randomization on agent generalization. Our experiments show that domain randomization may lead to suboptimal, high-variance policies, which we attribute to the uniform sampling of environment parameters. We propose Active Domain Randomization, a novel algorithm that learns a parameter sampling strategy. Our method looks for the most informative environment variations within the given randomization ranges by leveraging the discrepancies of policy rollouts in randomized and reference environment instances. We find that training more frequently on these instances leads to better overall agent generalization. Our experiments across various physics-based simulated and real-robot tasks show that this enhancement leads to more robust, consistent policies.

Code available at https://github.com/montrealrobotics/active-domainrand

References in corpus (9)

• Continuous control with deep reinforcement learning
• A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning
• Which Training Methods for GANs do actually Converge?
• Learning Dexterous In-Hand Manipulation
• Quantifying Generalization in Reinforcement Learning
• Assessing Generalization in Deep Reinforcement Learning
• An Empirical Study of Example Forgetting during Deep Neural Network Learning
• Generalization and Regularization in DQN
• Learning Self-Imitating Diverse Policies

Cited by in corpus (18)

• Solving Rubik's Cube with a Robot Hand
• BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning
• Learning of Parameters in Behavior Trees for Movement Skills
• Variational Automatic Curriculum Learning for Sparse-Reward Cooperative Multi-Agent Problems
• Robust Reinforcement Learning using Adversarial Populations
• ROBEL: Robotics Benchmarks for Learning with Low-Cost Robots
• Rapidly Adaptable Legged Robots via Evolutionary Meta-Learning
• Learning to Collide: An Adaptive Safety-Critical Scenarios Generating Method
• SimGAN: Hybrid Simulator Identification for Domain Adaptation via Adversarial Reinforcement Learning
• Measuring Visual Generalization in Continuous Control from Pixels
• Learning from scarce information: using synthetic data to classify Roman fine ware pottery
• Neural Data Server: A Large-Scale Search Engine for Transfer Learning Data
• Discovering Generalizable Skills via Automated Generation of Diverse Tasks
• Not Only Domain Randomization: Universal Policy with Embedding System Identification
• Robo-PlaNet: Learning to Poke in a Day
• Adaptation of Quadruped Robot Locomotion with Meta-Learning
• Training Lightweight CNNs for Human-Nanodrone Proximity Interaction from Small Datasets using Background Randomization
• Extrapolation in Gridworld Markov-Decision Processes