Why Generalization in RL is Difficult: Epistemic POMDPs and Implicit Partial Observability
arXiv:2107.06277
Abstract
Generalization is a central challenge for the deployment of reinforcement learning (RL) systems in the real world. In this paper, we show that the sequential structure of the RL problem necessitates new approaches to generalization beyond the well-studied techniques used in supervised learning. While supervised learning methods can generalize effectively without explicitly accounting for epistemic uncertainty, we show that, perhaps surprisingly, this is not the case in RL. We show that generalization to unseen test conditions from a limited number of training conditions induces implicit partial observability, effectively turning even fully-observed MDPs into POMDPs. Informed by this observation, we recast the problem of generalization in RL as solving the induced partially observed Markov decision process, which we call the epistemic POMDP. We demonstrate the failure modes of algorithms that do not appropriately handle this partial observability, and suggest a simple ensemble-based technique for approximately solving the partially observed problem. Empirically, we demonstrate that our simple algorithm derived from the epistemic POMDP achieves significant gains in generalization over current methods on the Procgen benchmark suite.
arXiv commentFirst two authors contributed equally
References in corpus (20)
- Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
- Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
- Solving Rubik's Cube with a Robot Hand
- Model-Based Bayesian Exploration
- (More) Efficient Reinforcement Learning via Posterior Sampling
- Leveraging Procedural Generation to Benchmark Reinforcement Learning
- Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels
- Illuminating Generalization in Deep Reinforcement Learning through Procedural Level Generation
- Generalization and Regularization in DQN
- A Dissection of Overfitting and Generalization in Continuous Reinforcement Learning
- Gotta Learn Fast: A New Benchmark for Generalization in RL
- Learning Invariant Representations for Reinforcement Learning without Reconstruction
- Decoupling Representation Learning from Reinforcement Learning
- VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning
- Generalization in Reinforcement Learning with Selective Noise Injection and Information Bottleneck
- Automatic Data Augmentation for Generalization in Deep Reinforcement Learning
- The NetHack Learning Environment
- If MaxEnt RL is the Answer, What is the Question?
- Dynamics Generalization via Information Bottleneck in Deep Reinforcement Learning
- Geometry and Determinism of Optimal Stationary Control in Partially Observable Markov Decision Processes