Observational Overfitting in Reinforcement Learning
arXiv:1912.02975
Abstract
A major component of overfitting in model-free reinforcement learning (RL) involves the case where the agent may mistakenly correlate reward with certain spurious features from the observations generated by the Markov Decision Process (MDP). We provide a general framework for analyzing this scenario, which we use to design multiple synthetic benchmarks from only modifying the observation space of an MDP. When an agent overfits to different observation spaces even if the underlying MDP dynamics is fixed, we term this observational overfitting. Our experiments expose intriguing properties especially with regards to implicit regularization, and also corroborate results from previous works in RL generalization and supervised learning (SL).
Published as a conference paper in ICLR 2020
References in corpus (23)
- Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
- Playing Atari with Deep Reinforcement Learning
- Neural Tangent Kernel: Convergence and Generalization in Neural Networks
- StarCraft II: A New Challenge for Reinforcement Learning
- Robust Adversarial Reinforcement Learning
- Invariant Risk Minimization
- Global Convergence of Policy Gradient Methods for the Linear Quadratic Regulator
- A Study on Overfitting in Deep Reinforcement Learning
- Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data
- Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networks
- Quantifying Generalization in Reinforcement Learning
- Visualizing and Understanding Atari Agents
- Simple random search provides a competitive approach to reinforcement learning
- Assessing Generalization in Deep Reinforcement Learning
- A Dissection of Overfitting and Generalization in Continuous Reinforcement Learning
- Gotta Learn Fast: A New Benchmark for Generalization in RL
- Implicit Regularization in Deep Learning
- Implicit Regularization in Deep Matrix Factorization
- Generalization in Reinforcement Learning with Selective Noise Injection and Information Bottleneck
- Generalization in Deep Networks: The Role of Distance from Initialization
- Understanding the impact of entropy on policy optimization
- Natural Environment Benchmarks for Reinforcement Learning
- On the Generalization Gap in Reparameterizable Reinforcement Learning
Cited by in corpus (6)
- A Survey of Zero-shot Generalisation in Deep Reinforcement Learning
- Automated Reinforcement Learning (AutoRL): A Survey and Open Problems
- Generalization in Reinforcement Learning by Soft Data Augmentation
- D2RL: Deep Dense Architectures in Reinforcement Learning
- Invariant Policy Optimization: Towards Stronger Generalization in Reinforcement Learning
- Instance based Generalization in Reinforcement Learning