Measuring Visual Generalization in Continuous Control from Pixels

Abstract

Self-supervised learning and data augmentation have significantly reduced the performance gap between state and image-based reinforcement learning agents in continuous control tasks. However, it is still unclear whether current techniques can face a variety of visual conditions required by real-world environments. We propose a challenging benchmark that tests agents' visual generalization by adding graphical variety to existing continuous control domains. Our empirical analysis shows that current methods struggle to generalize across a diverse set of visual changes, and we examine the specific factors of variation that make these tasks difficult. We find that data augmentation techniques outperform self-supervised learning approaches and that more significant image transformations provide better visual generalization \footnote{The benchmark and our augmented actor-critic implementation are open-sourced @ https://github.com/QData/dmc_remastered)

A total of 20 pages, 8 pages as the main text

References in corpus (20)

• Neural Architecture Search with Reinforcement Learning
• Improved Regularization of Convolutional Neural Networks with Cutout
• Addressing Function Approximation Error in Actor-Critic Methods
• Soft Actor-Critic Algorithms and Applications
• QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation
• Solving Rubik's Cube with a Robot Hand
• Efficient Neural Architecture Search via Parameter Sharing
• Data Augmentation using Random Image Cropping and Patching for Deep CNNs
• Reinforcement Learning with Augmented Data
• Quantifying Generalization in Reinforcement Learning
• Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Research
• Leveraging Procedural Generation to Benchmark Reinforcement Learning
• Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels
• Illuminating Generalization in Deep Reinforcement Learning through Procedural Level Generation
• Gotta Learn Fast: A New Benchmark for Generalization in RL
• Learning Invariant Representations for Reinforcement Learning without Reconstruction
• Investigating Generalisation in Continuous Deep Reinforcement Learning
• Active Domain Randomization
• Natural Environment Benchmarks for Reinforcement Learning
• Invariant Policy Optimization: Towards Stronger Generalization in Reinforcement Learning

Cited by in corpus (3)

• A Survey of Zero-shot Generalisation in Deep Reinforcement Learning
• Automatic Data Augmentation for Generalization in Deep Reinforcement Learning
• Decoupling Value and Policy for Generalization in Reinforcement Learning