A Study on Overfitting in Deep Reinforcement Learning

Abstract

Recent years have witnessed significant progresses in deep Reinforcement Learning (RL). Empowered with large scale neural networks, carefully designed architectures, novel training algorithms and massively parallel computing devices, researchers are able to attack many challenging RL problems. However, in machine learning, more training power comes with a potential risk of more overfitting. As deep RL techniques are being applied to critical problems such as healthcare and finance, it is important to understand the generalization behaviors of the trained agents. In this paper, we conduct a systematic study of standard RL agents and find that they could overfit in various ways. Moreover, overfitting could happen "robustly": commonly used techniques in RL that add stochasticity do not necessarily prevent or detect overfitting. In particular, the same agents and learning algorithms could have drastically different test performance, even when all of them achieve optimal rewards during training. The observations call for more principled and careful evaluation protocols in RL. We conclude with a general discussion on overfitting in RL and a study of the generalization behaviors from the perspective of inductive bias.

References in corpus (8)

• Playing Atari with Deep Reinforcement Learning
• Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm
• StarCraft II: A New Challenge for Reinforcement Learning
• Massively Parallel Methods for Deep Reinforcement Learning
• Train faster, generalize better: Stability of stochastic gradient descent
• Taming the Monster: A Fast and Simple Algorithm for Contextual Bandits
• Fisher-Rao Metric, Geometry, and Complexity of Neural Networks
• Can Deep Reinforcement Learning Solve Erdos-Selfridge-Spencer Games?

Cited by in corpus (80)

• An Introduction to Deep Reinforcement Learning
• An Application of Deep Reinforcement Learning to Algorithmic Trading
• Quantifying Generalization in Reinforcement Learning
• A Survey of Zero-shot Generalisation in Deep Reinforcement Learning
• Relational Deep Reinforcement Learning
• Dark, Beyond Deep: A Paradigm Shift to Cognitive AI with Humanlike Common Sense
• Scalable agent alignment via reward modeling: a research direction
• Illuminating Generalization in Deep Reinforcement Learning through Procedural Level Generation
• A Deep Generative Model for Molecule Optimization via One Fragment Modification
• Automated Reinforcement Learning (AutoRL): A Survey and Open Problems
• Q-Learning based system for path planning with unmanned aerial vehicles swarms in obstacle environments
• Learning to Generalize from Sparse and Underspecified Rewards
• Transfer Learning for Related Reinforcement Learning Tasks via Image-to-Image Translation
• Meta reinforcement learning as task inference
• On the Binding Problem in Artificial Neural Networks
• Network Randomization: A Simple Technique for Generalization in Deep Reinforcement Learning
• On quadrature rules for solving Partial Differential Equations using Neural Networks
• Domain Generalization with MixStyle
• Path integral contour deformations for observables in $SU(N)$ gauge theory
• Investigating Generalisation in Continuous Deep Reinforcement Learning
• RIDE: Rewarding Impact-Driven Exploration for Procedurally-Generated Environments
• Pretraining Representations for Data-Efficient Reinforcement Learning
• Maximum Entropy RL (Provably) Solves Some Robust RL Problems
• Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learning
• Human-like generalization in a machine through predicate learning
• A Theory of Relation Learning and Cross-domain Generalization
• Meta-Learning through Hebbian Plasticity in Random Networks
• Towards Practical Multi-Object Manipulation using Relational Reinforcement Learning
• Generalization in Transfer Learning
• Distributionally Robust Reinforcement Learning
• Distributional Robustness and Regularization in Reinforcement Learning
• Observational Overfitting in Reinforcement Learning
• Regularization Matters in Policy Optimization
• Balancing Profit, Risk, and Sustainability for Portfolio Management
• MixStyle Neural Networks for Domain Generalization and Adaptation
• Learning to Select Goals in Automated Planning with Deep-Q Learning
• Neural Logic Reinforcement Learning
• Evolving and Merging Hebbian Learning Rules: Increasing Generalization by Decreasing the Number of Rules
• Adversarially Guided Actor-Critic
• Interference and Generalization in Temporal Difference Learning
• Task-Agnostic Dynamics Priors for Deep Reinforcement Learning
• Transient Non-Stationarity and Generalisation in Deep Reinforcement Learning
• Generalizing from a few environments in safety-critical reinforcement learning
• One Solution is Not All You Need: Few-Shot Extrapolation via Structured MaxEnt RL
• Fast Adaptation via Policy-Dynamics Value Functions
• Improving the Generalization of End-to-End Driving through Procedural Generation
• Reinforcement Learning Generalization with Surprise Minimization
• Robust Visual Domain Randomization for Reinforcement Learning
• Domain Adversarial Reinforcement Learning
• Generalization of Reinforcement Learning with Policy-Aware Adversarial Data Augmentation
• Convolutional Neural Network(CNN/ConvNet) in Stock Price Movement Prediction
• Rank the Episodes: A Simple Approach for Exploration in Procedurally-Generated Environments
• Sample-efficient reinforcement learning using deep Gaussian processes
• Theory-based Causal Transfer: Integrating Instance-level Induction and Abstract-level Structure Learning
• Transfer Learning in Multi-Agent Reinforcement Learning with Double Q-Networks for Distributed Resource Sharing in V2X Communication
• A Survey on Reproducibility by Evaluating Deep Reinforcement Learning Algorithms on Real-World Robots
• Evolutionary Computation and AI Safety: Research Problems Impeding Routine and Safe Real-world Application of Evolution
• DRIBO: Robust Deep Reinforcement Learning via Multi-View Information Bottleneck
• Improving Generalization in Meta-RL with Imaginary Tasks from Latent Dynamics Mixture
• Off-Policy Meta-Reinforcement Learning Based on Feature Embedding Spaces
• Towards Deeper Deep Reinforcement Learning with Spectral Normalization
• Online Safety Assurance for Deep Reinforcement Learning
• Sparse Attention Guided Dynamic Value Estimation for Single-Task Multi-Scene Reinforcement Learning
• Dynamic Value Estimation for Single-Task Multi-Scene Reinforcement Learning
• A Brief Look at Generalization in Visual Meta-Reinforcement Learning
• Rogue-Gym: A New Challenge for Generalization in Reinforcement Learning
• Meta Arcade: A Configurable Environment Suite for Meta-Learning
• MAME : Model-Agnostic Meta-Exploration
• Goal Reasoning by Selecting Subgoals with Deep Q-Learning
• Analyzing Visual Representations in Embodied Navigation Tasks
• Dual-Arm Adversarial Robot Learning
• Extrapolation in Gridworld Markov-Decision Processes
• An investigation of model-free planning
• Transferring Deep Reinforcement Learning with Adversarial Objective and Augmentation
• Enhanced Scene Specificity with Sparse Dynamic Value Estimation
• Constructing Ancestral Recombination Graphs through Reinforcement Learning
• Learning Provably Robust Motion Planners Using Funnel Libraries
• Conditioning of Reinforcement Learning Agents and its Policy Regularization Application
• How to Train your Quadrotor: A Framework for Consistently Smooth and Responsive Flight Control via Reinforcement Learning
• The Impact of Activation Sparsity on Overfitting in Convolutional Neural Networks