A Dissection of Overfitting and Generalization in Continuous Reinforcement Learning

Abstract

The risks and perils of overfitting in machine learning are well known. However most of the treatment of this, including diagnostic tools and remedies, was developed for the supervised learning case. In this work, we aim to offer new perspectives on the characterization and prevention of overfitting in deep Reinforcement Learning (RL) methods, with a particular focus on continuous domains. We examine several aspects, such as how to define and diagnose overfitting in MDPs, and how to reduce risks by injecting sufficient training diversity. This work complements recent findings on the brittleness of deep RL methods and offers practical observations for RL researchers and practitioners.

20 pages, 16 figures

References in corpus (2)

• Adam: A Method for Stochastic Optimization
• DeepMind Lab

Cited by in corpus (48)

• An Introduction to Deep Reinforcement Learning
• MOPO: Model-based Offline Policy Optimization
• Quantifying Generalization in Reinforcement Learning
• A Survey of Zero-shot Generalisation in Deep Reinforcement Learning
• Leveraging Procedural Generation to Benchmark Reinforcement Learning
• Assessing Generalization in Deep Reinforcement Learning
• Q-Learning based system for path planning with unmanned aerial vehicles swarms in obstacle environments
• Automatic Data Augmentation for Generalization in Deep Reinforcement Learning
• Network Randomization: A Simple Technique for Generalization in Deep Reinforcement Learning
• Learning with AMIGo: Adversarially Motivated Intrinsic Goals
• Learning to Locomote: Understanding How Environment Design Matters for Deep Reinforcement Learning
• Investigating Generalisation in Continuous Deep Reinforcement Learning
• RIDE: Rewarding Impact-Driven Exploration for Procedurally-Generated Environments
• Generalizable control for multiparameter quantum metrology
• Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learning
• Off-Policy Evaluation via Off-Policy Classification
• Analysing Deep Reinforcement Learning Agents Trained with Domain Randomisation
• Observational Overfitting in Reinforcement Learning
• Regularization Matters in Policy Optimization
• SECANT: Self-Expert Cloning for Zero-Shot Generalization of Visual Policies
• Evolving and Merging Hebbian Learning Rules: Increasing Generalization by Decreasing the Number of Rules
• Adversarially Guided Actor-Critic
• CARLA Real Traffic Scenarios -- novel training ground and benchmark for autonomous driving
• Why Generalization in RL is Difficult: Epistemic POMDPs and Implicit Partial Observability
• Transient Non-Stationarity and Generalisation in Deep Reinforcement Learning
• Interference and Generalization in Temporal Difference Learning
• Fast Adaptation via Policy-Dynamics Value Functions
• Continuous Coordination As a Realistic Scenario for Lifelong Learning
• Robust Visual Domain Randomization for Reinforcement Learning
• When Is Generalizable Reinforcement Learning Tractable?
• Generalization of Reinforcement Learning with Policy-Aware Adversarial Data Augmentation
• Rank the Episodes: A Simple Approach for Exploration in Procedurally-Generated Environments
• Robust Domain Randomised Reinforcement Learning through Peer-to-Peer Distillation
• Analysis and Improvement of Adversarial Training in DQN Agents With Adversarially-Guided Exploration (AGE)
• MRAC-RL: A Framework for On-Line Policy Adaptation Under Parametric Model Uncertainty
• Decoupling Value and Policy for Generalization in Reinforcement Learning
• Multimodal Reward Shaping for Efficient Exploration in Reinforcement Learning
• Recurrent Value Functions
• Sparse Attention Guided Dynamic Value Estimation for Single-Task Multi-Scene Reinforcement Learning
• A Brief Look at Generalization in Visual Meta-Reinforcement Learning
• The Principle of Unchanged Optimality in Reinforcement Learning Generalization
• Towards Deeper Deep Reinforcement Learning with Spectral Normalization
• Replay-Guided Adversarial Environment Design
• Towards robust and domain agnostic reinforcement learning competitions
• Enhanced Scene Specificity with Sparse Dynamic Value Estimation
• How to Train your Quadrotor: A Framework for Consistently Smooth and Responsive Flight Control via Reinforcement Learning
• Improved Memories Learning
• An investigation of model-free planning