Generalization and Regularization in DQN

Abstract

Deep reinforcement learning algorithms have shown an impressive ability to learn complex control policies in high-dimensional tasks. However, despite the ever-increasing performance on popular benchmarks, policies learned by deep reinforcement learning algorithms can struggle to generalize when evaluated in remarkably similar environments. In this paper we propose a protocol to evaluate generalization in reinforcement learning through different modes of Atari 2600 games. With that protocol we assess the generalization capabilities of DQN, one of the most traditional deep reinforcement learning algorithms, and we provide evidence suggesting that DQN overspecializes to the training environment. We then comprehensively evaluate the impact of dropout and ${\ell }_2$ regularization, as well as the impact of reusing learned representations to improve the generalization capabilities of DQN. Despite regularization being largely underutilized in deep reinforcement learning, we show that it can, in fact, help DQN learn more general features. These features can be reused and fine-tuned on similar tasks, considerably improving DQN's sample efficiency.

arXiv commentEarlier versions of this work were presented both at the NeurIPS'18 Deep Reinforcement Learning Workshop and the 4th Multidisciplinary Conference on Reinforcement Learning and Decision Making (RLDM'19)

References in corpus (6)

• Overcoming catastrophic forgetting in neural networks
• How transferable are features in deep neural networks?
• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
• Actor-Mimic: Deep Multitask and Transfer Reinforcement Learning
• Quantifying Generalization in Reinforcement Learning
• Gotta Learn Fast: A New Benchmark for Generalization in RL

Cited by in corpus (49)

• Quantifying Generalization in Reinforcement Learning
• A Survey of Zero-shot Generalisation in Deep Reinforcement Learning
• Leveraging Procedural Generation to Benchmark Reinforcement Learning
• Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels
• Automatic Data Augmentation for Generalization in Deep Reinforcement Learning
• Network Randomization: A Simple Technique for Generalization in Deep Reinforcement Learning
• Domain Generalization with MixStyle
• Stabilizing Deep Q-Learning with ConvNets and Vision Transformers under Data Augmentation
• Active Domain Randomization
• Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learning
• CybORG: An Autonomous Cyber Operations Research Gym
• Deep Reinforcement and InfoMax Learning
• Strategies for Using Proximal Policy Optimization in Mobile Puzzle Games
• Regularization Matters in Policy Optimization
• MixStyle Neural Networks for Domain Generalization and Adaptation
• Learning Task-relevant Representations for Generalization via Characteristic Functions of Reward Sequence Distributions
• SECANT: Self-Expert Cloning for Zero-Shot Generalization of Visual Policies
• Robust Predictable Control
• F2A2: Flexible Fully-decentralized Approximate Actor-critic for Cooperative Multi-agent Reinforcement Learning
• Adversarially Guided Actor-Critic
• Why Generalization in RL is Difficult: Epistemic POMDPs and Implicit Partial Observability
• Interference and Generalization in Temporal Difference Learning
• One Solution is Not All You Need: Few-Shot Extrapolation via Structured MaxEnt RL
• Generalizing from a few environments in safety-critical reinforcement learning
• Reinforcement Learning Generalization with Surprise Minimization
• Return-based Scaling: Yet Another Normalisation Trick for Deep RL
• Robust Visual Domain Randomization for Reinforcement Learning
• When Is Generalizable Reinforcement Learning Tractable?
• Reasoning and Generalization in RL: A Tool Use Perspective
• Joint Attention for Multi-Agent Coordination and Social Learning
• Sequoia: A Software Framework to Unify Continual Learning Research
• Decoupling Value and Policy for Generalization in Reinforcement Learning
• On Catastrophic Interference in Atari 2600 Games
• RAPID: Enabling Fast Online Policy Learning in Dynamic Public Cloud Environments
• DRIBO: Robust Deep Reinforcement Learning via Multi-View Information Bottleneck
• Group Equivariant Deep Reinforcement Learning
• Towards Deeper Deep Reinforcement Learning with Spectral Normalization
• A Brief Look at Generalization in Visual Meta-Reinforcement Learning
• Towards a practical measure of interference for reinforcement learning
• Correcting Momentum in Temporal Difference Learning
• Deep Reinforcement Learning with Weighted Q-Learning
• Improve Agents without Retraining: Parallel Tree Search with Off-Policy Correction
• Safety Enhancement for Deep Reinforcement Learning in Autonomous Separation Assurance
• Rogue-Gym: A New Challenge for Generalization in Reinforcement Learning
• On The Transferability of Deep-Q Networks
• In Hindsight: A Smooth Reward for Steady Exploration
• Conditioning of Reinforcement Learning Agents and its Policy Regularization Application
• Efficient decorrelation of features using Gramian in Reinforcement Learning
• Learning robust driving policies without online exploration