Illuminating Generalization in Deep Reinforcement Learning through Procedural Level Generation

Abstract

Deep reinforcement learning (RL) has shown impressive results in a variety of domains, learning directly from high-dimensional sensory streams. However, when neural networks are trained in a fixed environment, such as a single level in a video game, they will usually overfit and fail to generalize to new levels. When RL models overfit, even slight modifications to the environment can result in poor agent performance. This paper explores how procedurally generated levels during training can increase generality. We show that for some games procedural level generation enables generalization to new levels within the same distribution. Additionally, it is possible to achieve better performance with less data by manipulating the difficulty of the levels in response to the performance of the agent. The generality of the learned behaviors is also evaluated on a set of human-designed levels. The results suggest that the ability to generalize to human-designed levels highly depends on the design of the level generators. We apply dimensionality reduction and clustering techniques to visualize the generators' distributions of levels and analyze to what degree they can produce levels similar to those designed by a human.

arXiv commentAccepted to NeurIPS Deep RL Workshop 2018

References in corpus (8)

• An overview of gradient descent optimization algorithms
• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
• Hindsight Experience Replay
• A Study on Overfitting in Deep Reinforcement Learning
• Schema Networks: Zero-shot Transfer with a Generative Causal Model of Intuitive Physics
• Reverse Curriculum Generation for Reinforcement Learning
• Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Play
• Automated Curriculum Learning for Neural Networks

Cited by in corpus (51)

• Unity: A General Platform for Intelligent Agents
• Quantifying Generalization in Reinforcement Learning
• Leveraging Procedural Generation to Benchmark Reinforcement Learning
• Procedural Content Generation through Quality Diversity
• Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutions
• Assessing Generalization in Deep Reinforcement Learning
• Open-Ended Learning Leads to Generally Capable Agents
• Automatic Data Augmentation for Generalization in Deep Reinforcement Learning
• AI-GAs: AI-generating algorithms, an alternate paradigm for producing general artificial intelligence
• The NetHack Learning Environment
• Enhanced POET: Open-Ended Reinforcement Learning through Unbounded Invention of Learning Challenges and their Solutions
• Domain Generalization with MixStyle
• RIDE: Rewarding Impact-Driven Exploration for Procedurally-Generated Environments
• Maximum Entropy RL (Provably) Solves Some Robust RL Problems
• Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learning
• Meta-Learning through Hebbian Plasticity in Random Networks
• Towards Practical Multi-Object Manipulation using Relational Reinforcement Learning
• Multi-Robot Coordination and Layout Design for Automated Warehousing
• Strategies for Using Proximal Policy Optimization in Mobile Puzzle Games
• Procedural Content Generation using Neuroevolution and Novelty Search for Diverse Video Game Levels
• MixStyle Neural Networks for Domain Generalization and Adaptation
• Evolving and Merging Hebbian Learning Rules: Increasing Generalization by Decreasing the Number of Rules
• SECANT: Self-Expert Cloning for Zero-Shot Generalization of Visual Policies
• Reinforcement Learning with Dual-Observation for General Video Game Playing
• Adversarially Guided Actor-Critic
• Diverse Multimedia Layout Generation with Multi Choice Learning
• Procedural Content Generation: Better Benchmarks for Transfer Reinforcement Learning
• Interference and Generalization in Temporal Difference Learning
• On the Measure of Intelligence
• Why Generalization in RL is Difficult: Epistemic POMDPs and Implicit Partial Observability
• Rotation, Translation, and Cropping for Zero-Shot Generalization
• Measuring Visual Generalization in Continuous Control from Pixels
• Agent-Based Adaptive Level Generation for Dynamic Difficulty Adjustment in Angry Birds
• Meta Automatic Curriculum Learning
• Procedural Generalization by Planning with Self-Supervised World Models
• TeachMyAgent: a Benchmark for Automatic Curriculum Learning in Deep RL
• Decoupling Value and Policy for Generalization in Reinforcement Learning
• Prioritized Level Replay
• Adaptive Procedural Task Generation for Hard-Exploration Problems
• Scenic4RL: Programmatic Modeling and Generation of Reinforcement Learning Environments
• Evolutionary Computation and AI Safety: Research Problems Impeding Routine and Safe Real-world Application of Evolution
• Testing the Genomic Bottleneck Hypothesis in Hebbian Meta-Learning
• Sparse Attention Guided Dynamic Value Estimation for Single-Task Multi-Scene Reinforcement Learning
• Learning Controllable Content Generators
• Dynamic Value Estimation for Single-Task Multi-Scene Reinforcement Learning
• MimicBot: Combining Imitation and Reinforcement Learning to win in Bot Bowl
• Curriculum Learning for Deep Generative Models with Clustering
• A Brief Look at Generalization in Visual Meta-Reinforcement Learning
• Tree Search vs Optimization Approaches for Map Generation
• Enhanced Scene Specificity with Sparse Dynamic Value Estimation
• Fixed $\beta$-VAE Encoding for Curious Exploration in Complex 3D Environments