Instance based Generalization in Reinforcement Learning
arXiv:2011.01089
Abstract
Agents trained via deep reinforcement learning (RL) routinely fail to generalize to unseen environments, even when these share the same underlying dynamics as the training levels. Understanding the generalization properties of RL is one of the challenges of modern machine learning. Towards this goal, we analyze policy learning in the context of Partially Observable Markov Decision Processes (POMDPs) and formalize the dynamics of training levels as instances. We prove that, independently of the exploration strategy, reusing instances introduces significant changes on the effective Markov dynamics the agent observes during training. Maximizing expected rewards impacts the learned belief state of the agent by inducing undesired instance specific speedrunning policies instead of generalizeable ones, which are suboptimal on the training set. We provide generalization bounds to the value gap in train and test environments based on the number of training instances, and use insights based on these to improve performance on unseen levels. We propose training a shared belief representation over an ensemble of specialized policies, from which we compute a consensus policy that is used for data collection, disallowing instance specific exploitation. We experimentally validate our theory, observations, and the proposed computational solution over the CoinRun benchmark.
arXiv commentAccepted on NeurIPS 2020
References in corpus (18)
- Playing Atari with Deep Reinforcement Learning
- Improved Regularization of Convolutional Neural Networks with Cutout
- Reconciling modern machine learning practice and the bias-variance trade-off
- Understanding deep learning requires rethinking generalization
- IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
- Value-Function Approximations for Partially Observable Markov Decision Processes
- Rainbow: Combining Improvements in Deep Reinforcement Learning
- Agent57: Outperforming the Atari Human Benchmark
- Quantifying Generalization in Reinforcement Learning
- Leveraging Procedural Generation to Benchmark Reinforcement Learning
- TextWorld: A Learning Environment for Text-based Games
- BabyAI: A Platform to Study the Sample Efficiency of Grounded Language Learning
- Generalization in Reinforcement Learning with Selective Noise Injection and Information Bottleneck
- InfoBot: Transfer and Exploration via the Information Bottleneck
- Neural Predictive Belief Representations
- Particle Filter Networks with Application to Visual Localization
- Observational Overfitting in Reinforcement Learning
- Interactive Language Learning by Question Answering