Improving Generalization in Reinforcement Learning with Mixture Regularization

Abstract

Deep reinforcement learning (RL) agents trained in a limited set of environments tend to suffer overfitting and fail to generalize to unseen testing environments. To improve their generalizability, data augmentation approaches (e.g. cutout and random convolution) are previously explored to increase the data diversity. However, we find these approaches only locally perturb the observations regardless of the training environments, showing limited effectiveness on enhancing the data diversity and the generalization performance. In this work, we introduce a simple approach, named mixreg, which trains agents on a mixture of observations from different training environments and imposes linearity constraints on the observation interpolations and the supervision (e.g. associated reward) interpolations. Mixreg increases the data diversity more effectively and helps learn smoother policies. We verify its effectiveness on improving generalization by conducting extensive experiments on the large-scale Procgen benchmark. Results show mixreg outperforms the well-established baselines on unseen testing environments by a large margin. Mixreg is simple, effective and general. It can be applied to both policy-based and value-based RL algorithms. Code is available at https://github.com/kaixin96/mixreg .

arXiv commentNeurIPS 2020

References in corpus (10)

• mixup: Beyond Empirical Risk Minimization
• Trust Region Policy Optimization
• Dueling Network Architectures for Deep Reinforcement Learning
• High-Dimensional Continuous Control Using Generalized Advantage Estimation
• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
• Noisy Networks for Exploration
• Rainbow: Combining Improvements in Deep Reinforcement Learning
• Reinforcement Learning with Augmented Data
• Quantifying Generalization in Reinforcement Learning
• Leveraging Procedural Generation to Benchmark Reinforcement Learning

Cited by in corpus (8)

• A Survey of Zero-shot Generalisation in Deep Reinforcement Learning
• Automatic Data Augmentation for Generalization in Deep Reinforcement Learning
• Deep Reinforcement Learning at the Edge of the Statistical Precipice
• Stabilizing Deep Q-Learning with ConvNets and Vision Transformers under Data Augmentation
• SECANT: Self-Expert Cloning for Zero-Shot Generalization of Visual Policies
• CCLF: A Contrastive-Curiosity-Driven Learning Framework for Sample-Efficient Reinforcement Learning
• Generalization of Reinforcement Learning with Policy-Aware Adversarial Data Augmentation
• Decoupling Value and Policy for Generalization in Reinforcement Learning