Network Randomization: A Simple Technique for Generalization in Deep Reinforcement Learning

Abstract

Deep reinforcement learning (RL) agents often fail to generalize to unseen environments (yet semantically similar to trained agents), particularly when they are trained on high-dimensional state spaces, such as images. In this paper, we propose a simple technique to improve a generalization ability of deep RL agents by introducing a randomized (convolutional) neural network that randomly perturbs input observations. It enables trained agents to adapt to new domains by learning robust features invariant across varied and randomized environments. Furthermore, we consider an inference method based on the Monte Carlo approximation to reduce the variance induced by this randomization. We demonstrate the superiority of our method across 2D CoinRun, 3D DeepMind Lab exploration and 3D robotics control tasks: it significantly outperforms various regularization and data augmentation methods for the same purpose.

Accepted in ICLR 2020 and NeurIPS Workshop on Deep RL 2019 / First two authors are equally contributed

References in corpus (19)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Trust Region Policy Optimization
• Improved Regularization of Convolutional Neural Networks with Cutout
• Unsupervised Data Augmentation for Consistency Training
• AutoAugment: Learning Augmentation Policies from Data
• StarCraft II: A New Challenge for Reinforcement Learning
• ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness
• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
• Learning to reinforcement learn
• A Study on Overfitting in Deep Reinforcement Learning
• Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World
• DARLA: Improving Zero-Shot Transfer in Reinforcement Learning
• Assessing Generalization in Deep Reinforcement Learning
• Generalization and Regularization in DQN
• Tactics of Adversarial Attack on Deep Reinforcement Learning Agents
• A Dissection of Overfitting and Generalization in Continuous Reinforcement Learning
• Policy Distillation
• Environment Probing Interaction Policies
• Domain Randomization for Active Pose Estimation

Cited by in corpus (12)

• Domain Generalization: A Survey
• A Survey of Zero-shot Generalisation in Deep Reinforcement Learning
• Leveraging Procedural Generation to Benchmark Reinforcement Learning
• A review on Deep Reinforcement Learning for Fluid Mechanics
• A review on deep reinforcement learning for fluid mechanics: an update
• GuidedMixup: An Efficient Mixup Strategy Guided by Saliency Maps
• Learning Task-relevant Representations for Generalization via Characteristic Functions of Reward Sequence Distributions
• CCLF: A Contrastive-Curiosity-Driven Learning Framework for Sample-Efficient Reinforcement Learning
• Rotation, Translation, and Cropping for Zero-Shot Generalization
• Robust Visual Domain Randomization for Reinforcement Learning
• PAC-Bayesian Soft Actor-Critic Learning
• Curriculum-based Sensing Reduction in Simulation to Real-World Transfer for In-hand Manipulation