Generalization in Reinforcement Learning by Soft Data Augmentation

Abstract

Extensive efforts have been made to improve the generalization ability of Reinforcement Learning (RL) methods via domain randomization and data augmentation. However, as more factors of variation are introduced during training, optimization becomes increasingly challenging, and empirically may result in lower sample efficiency and unstable training. Instead of learning policies directly from augmented data, we propose SOft Data Augmentation (SODA), a method that decouples augmentation from policy learning. Specifically, SODA imposes a soft constraint on the encoder that aims to maximize the mutual information between latent representations of augmented and non-augmented data, while the RL optimization process uses strictly non-augmented data. Empirical evaluations are performed on diverse tasks from DeepMind Control suite as well as a robotic manipulation task, and we find SODA to significantly advance sample efficiency, generalization, and stability in training over state-of-the-art vision-based RL methods.

Website: https://nicklashansen.github.io/SODA/ Code: https://github.com/nicklashansen/dmcontrol-generalization-benchmark. Presented at International Conference on Robotics and Automation (ICRA) 2021

References in corpus (17)

• A Simple Framework for Contrastive Learning of Visual Representations
• Playing Atari with Deep Reinforcement Learning
• Bootstrap your own latent: A new approach to self-supervised Learning
• QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation
• Contrastive Multiview Coding
• DeepMind Control Suite
• Reinforcement Learning with Augmented Data
• Quantifying Generalization in Reinforcement Learning
• Leveraging Procedural Generation to Benchmark Reinforcement Learning
• Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels
• Assessing Generalization in Deep Reinforcement Learning
• Decoupling Representation Learning from Reinforcement Learning
• Self-Supervised Policy Adaptation during Deployment
• Automatic Data Augmentation for Generalization in Deep Reinforcement Learning
• Data-Efficient Reinforcement Learning with Self-Predictive Representations
• Single Episode Policy Transfer in Reinforcement Learning
• Observational Overfitting in Reinforcement Learning

Cited by in corpus (5)

• A Survey of Zero-shot Generalisation in Deep Reinforcement Learning
• SECANT: Self-Expert Cloning for Zero-Shot Generalization of Visual Policies
• Scalable Bayesian Meta-Learning through Generalized Implicit Gradients
• Unsupervised Visual Attention and Invariance for Reinforcement Learning
• Equivariant $Q$ Learning in Spatial Action Spaces