SECANT: Self-Expert Cloning for Zero-Shot Generalization of Visual Policies
arXiv:2106.09678
Abstract
Generalization has been a long-standing challenge for reinforcement learning (RL). Visual RL, in particular, can be easily distracted by irrelevant factors in high-dimensional observation space. In this work, we consider robust policy learning which targets zero-shot generalization to unseen visual environments with large distributional shift. We propose SECANT, a novel self-expert cloning technique that leverages image augmentation in two stages to decouple robust representation learning from policy optimization. Specifically, an expert policy is first trained by RL from scratch with weak augmentations. A student network then learns to mimic the expert policy by supervised learning with strong augmentations, making its representation more robust against visual variations compared to the expert. Extensive experiments demonstrate that SECANT significantly advances the state of the art in zero-shot generalization across 4 challenging domains. Our average reward improvements over prior SOTAs are: DeepMind Control (+26.5%), robotic manipulation (+337.8%), vision-based autonomous driving (+47.7%), and indoor object navigation (+15.8%). Code release and video are available at https://linxifan.github.io/secant-site/.
arXiv commentICML 2021. Website: https://linxifan.github.io/secant-site/
References in corpus (39)
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- Distilling the Knowledge in a Neural Network
- Playing Atari with Deep Reinforcement Learning
- mixup: Beyond Empirical Risk Minimization
- FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence
- Soft Actor-Critic Algorithms and Applications
- MixMatch: A Holistic Approach to Semi-Supervised Learning
- DeepMind Control Suite
- Learning to reinforcement learn
- Generative Adversarial Imitation Learning
- CARLA: An Open Urban Driving Simulator
- Reinforcement Learning with Augmented Data
- Quantifying Generalization in Reinforcement Learning
- Leveraging Procedural Generation to Benchmark Reinforcement Learning
- ReMixMatch: Semi-Supervised Learning with Distribution Alignment and Augmentation Anchoring
- robosuite: A Modular Simulation Framework and Benchmark for Robot Learning
- Visualizing and Understanding Atari Agents
- Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels
- Illuminating Generalization in Deep Reinforcement Learning through Procedural Level Generation
- Assessing Generalization in Deep Reinforcement Learning
- Benchmarking Neural Network Robustness to Common Corruptions and Surface Variations
- Generalization and Regularization in DQN
- A Dissection of Overfitting and Generalization in Continuous Reinforcement Learning
- Learning Invariant Representations for Reinforcement Learning without Reconstruction
- Decoupling Representation Learning from Reinforcement Learning
- Self-Supervised Policy Adaptation during Deployment
- Robot Learning in Homes: Improving Generalization and Reducing Dataset Bias
- Generalization in Reinforcement Learning with Selective Noise Injection and Information Bottleneck
- Automatic Data Augmentation for Generalization in Deep Reinforcement Learning
- Improving Generalization in Reinforcement Learning with Mixture Regularization
- iGibson 1.0: a Simulation Environment for Interactive Tasks in Large Realistic Scenes
- Distilling Policy Distillation
- RIDE: Rewarding Impact-Driven Exploration for Procedurally-Generated Environments
- Single Episode Policy Transfer in Reinforcement Learning
- Generalization in Reinforcement Learning by Soft Data Augmentation
- On the Generalization Gap in Reparameterizable Reinforcement Learning
- Domain Adaptation Through Task Distillation
- Multi-task Learning for Continuous Control
- Prioritized Level Replay