One Solution is Not All You Need: Few-Shot Extrapolation via Structured MaxEnt RL
arXiv:2010.14484
Abstract
While reinforcement learning algorithms can learn effective policies for complex tasks, these policies are often brittle to even minor task variations, especially when variations are not explicitly provided during training. One natural approach to this problem is to train agents with manually specified variation in the training task or environment. However, this may be infeasible in practical situations, either because making perturbations is not possible, or because it is unclear how to choose suitable perturbation strategies without sacrificing performance. The key insight of this work is that learning diverse behaviors for accomplishing a task can directly lead to behavior that generalizes to varying environments, without needing to perform explicit perturbations during training. By identifying multiple solutions for the task in a single environment during training, our approach can generalize to new situations by abandoning solutions that are no longer effective and adopting those that are. We theoretically characterize a robustness set of environments that arises from our algorithm and empirically find that our diversity-driven approach can extrapolate to various changes in the environment and task.
arXiv commentAccepted at NeurIPS 2020
References in corpus (26)
- Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
- Playing Atari with Deep Reinforcement Learning
- Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
- RL: Fast Reinforcement Learning via Slow Reinforcement Learning
- Robust Adversarial Reinforcement Learning
- A Study on Overfitting in Deep Reinforcement Learning
- Uncertainty-Aware Reinforcement Learning for Collision Avoidance
- Quantifying Generalization in Reinforcement Learning
- Variational Intrinsic Control
- Meta-Reinforcement Learning of Structured Exploration Strategies
- EPOpt: Learning Robust Neural Network Policies Using Model Ensembles
- Generalization and Regularization in DQN
- Adversarial Policies: Attacking Deep Reinforcement Learning
- Neural probabilistic motor primitives for humanoid control
- Dynamics-Aware Unsupervised Discovery of Skills
- Action Robust Reinforcement Learning and Applications in Continuous Control
- VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning
- Improving Generalization in Meta Reinforcement Learning using Learned Objectives
- Generalization in Reinforcement Learning with Selective Noise Injection and Information Bottleneck
- Unsupervised Meta-Learning for Reinforcement Learning
- Preparing for the Unknown: Learning a Universal Policy with Online System Identification
- Worst Cases Policy Gradients
- Meta-Q-Learning
- Unsupervised Curricula for Visual Meta-Reinforcement Learning
- Reinforcement Learning with Perturbed Rewards
- VFunc: a Deep Generative Model for Functions