The Principle of Unchanged Optimality in Reinforcement Learning Generalization
arXiv:1906.00336
Abstract
Several recent papers have examined generalization in reinforcement learning (RL), by proposing new environments or ways to add noise to existing environments, then benchmarking algorithms and model architectures on those environments. We discuss subtle conceptual properties of RL benchmarks that are not required in supervised learning (SL), and also properties that an RL benchmark should possess. Chief among them is one we call the principle of unchanged optimality: there should exist a single that is optimal across all train and test tasks. In this work, we argue why this principle is important, and ways it can be broken or satisfied due to subtle choices in state representation or model architecture. We conclude by discussing challenges and future lines of research in theoretically analyzing generalization benchmarks.
arXiv commentPublished at ICML 2019 Workshop "Understanding and Improving Generalization in Deep Learning"
References in corpus (10)
- mixup: Beyond Empirical Risk Minimization
- Model-Based Reinforcement Learning for Atari
- Adversarial Examples Are Not Bugs, They Are Features
- Robust Adversarial Reinforcement Learning
- Visualizing and Understanding Atari Agents
- Unsupervised Predictive Memory in a Goal-Directed Agent
- A Dissection of Overfitting and Generalization in Continuous Reinforcement Learning
- Gotta Learn Fast: A New Benchmark for Generalization in RL
- Natural Environment Benchmarks for Reinforcement Learning
- On the Generalization Gap in Reparameterizable Reinforcement Learning