Procedural Generalization by Planning with Self-Supervised World Models
arXiv:2111.01587
Abstract
One of the key promises of model-based reinforcement learning is the ability to generalize using an internal model of the world to make predictions in novel environments and tasks. However, the generalization ability of model-based agents is not well understood because existing work has focused on model-free agents when benchmarking generalization. Here, we explicitly measure the generalization ability of model-based agents in comparison to their model-free counterparts. We focus our analysis on MuZero (Schrittwieser et al., 2020), a powerful model-based agent, and evaluate its performance on both procedural and task generalization. We identify three factors of procedural generalization -- planning, self-supervised representation learning, and procedural data diversity -- and show that by combining these techniques, we achieve state-of-the art generalization performance and data efficiency on Procgen (Cobbe et al., 2019). However, we find that these factors do not always provide the same benefits for the task generalization benchmarks in Meta-World (Yu et al., 2019), indicating that transfer remains a challenge and may require different approaches than procedural generalization. Overall, we suggest that building generalizable agents requires moving beyond the single-task, model-free paradigm and towards self-supervised model-based agents that are trained in rich, procedural, multi-task environments.
References in corpus (19)
- Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
- Recurrent World Models Facilitate Policy Evolution
- Learning to reinforcement learn
- Learning Continuous Control Policies by Stochastic Value Gradients
- Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning
- Benchmarking Model-Based Reinforcement Learning
- Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables
- Relational Deep Reinforcement Learning
- Illuminating Generalization in Deep Reinforcement Learning through Procedural Level Generation
- Gotta Learn Fast: A New Benchmark for Generalization in RL
- Improving Generalization in Meta Reinforcement Learning using Learned Objectives
- Generalization in Reinforcement Learning with Selective Noise Injection and Information Bottleneck
- Neural Predictive Belief Representations
- Monte-Carlo Tree Search as Regularized Policy Optimization
- Natural Environment Benchmarks for Reinforcement Learning
- Online and Offline Reinforcement Learning by Planning with a Learned Model
- Causally Correct Partial Models for Reinforcement Learning
- CoBERL: Contrastive BERT for Reinforcement Learning
- Cross-Trajectory Representation Learning for Zero-Shot Generalization in RL