Procedural Content Generation: Better Benchmarks for Transfer Reinforcement Learning
arXiv:2105.14780 · doi:10.1109/CoG52621.2021.9619000
Abstract
The idea of transfer in reinforcement learning (TRL) is intriguing: being able to transfer knowledge from one problem to another problem without learning everything from scratch. This promises quicker learning and learning more complex methods. To gain an insight into the field and to detect emerging trends, we performed a database search. We note a surprisingly late adoption of deep learning that starts in 2018. The introduction of deep learning has not yet solved the greatest challenge of TRL: generalization. Transfer between different domains works well when domains have strong similarities (e.g. MountainCar to Cartpole), and most TRL publications focus on different tasks within the same domain that have few differences. Most TRL applications we encountered compare their improvements against self-defined baselines, and the field is still missing unified benchmarks. We consider this to be a disappointing situation. For the future, we note that: (1) A clear measure of task similarity is needed. (2) Generalization needs to improve. Promising approaches merge deep learning with planning via MCTS or introduce memory through LSTMs. (3) The lack of benchmarking tools will be remedied to enable meaningful comparison and measure progress. Already Alchemy and Meta-World are emerging as interesting benchmark suites. We note that another development, the increase in procedural content generation (PCG), can improve both benchmarking and generalization in TRL.
References in corpus (20)
- Fast unfolding of communities in large networks
- Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model
- Dota 2 with Large Scale Deep Reinforcement Learning
- Benchmarking Deep Reinforcement Learning for Continuous Control
- RL: Fast Reinforcement Learning via Slow Reinforcement Learning
- Emergent Tool Use From Multi-Agent Autocurricula
- Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning
- Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey
- First return, then explore
- Leveraging Procedural Generation to Benchmark Reinforcement Learning
- Transfer Learning in Deep Reinforcement Learning: A Survey
- Carbon Emissions and Large Neural Network Training
- Illuminating Generalization in Deep Reinforcement Learning through Procedural Level Generation
- Assessing Transferability from Simulation to Reality for Reinforcement Learning
- Transfer Learning for Related Reinforcement Learning Tasks via Image-to-Image Translation
- Learning to Search with MCTSnets
- Single Episode Policy Transfer in Reinforcement Learning
- Continual Reinforcement Learning deployed in Real-life using Policy Distillation and Sim2Real Transfer
- MULTIPOLAR: Multi-Source Policy Aggregation for Transfer Reinforcement Learning between Diverse Environmental Dynamics
- A Distributed Reinforcement Learning Solution With Knowledge Transfer Capability for A Bike Rebalancing Problem