Investigating Generalisation in Continuous Deep Reinforcement Learning

Abstract

Deep Reinforcement Learning has shown great success in a variety of control tasks. However, it is unclear how close we are to the vision of putting Deep RL into practice to solve real world problems. In particular, common practice in the field is to train policies on largely deterministic simulators and to evaluate algorithms through training performance alone, without a train/test distinction to ensure models generalise and are not overfitted. Moreover, it is not standard practice to check for generalisation under domain shift, although robustness to such system change between training and testing would be necessary for real-world Deep RL control, for example, in robotics. In this paper we study these issues by first characterising the sources of uncertainty that provide generalisation challenges in Deep RL. We then provide a new benchmark and thorough empirical evaluation of generalisation challenges for state of the art Deep RL methods. In particular, we show that, if generalisation is the goal, then common practice of evaluating algorithms based on their training performance leads to the wrong conclusions about algorithm choice. Finally, we evaluate several techniques for improving generalisation and draw conclusions about the most robust techniques to date.

References in corpus (9)

• Benchmarking Deep Reinforcement Learning for Continuous Control
• DeepMind Control Suite
• Robust Adversarial Reinforcement Learning
• Deep Reinforcement Learning that Matters
• A Study on Overfitting in Deep Reinforcement Learning
• Quantifying Generalization in Reinforcement Learning
• Assessing Generalization in Deep Reinforcement Learning
• A Dissection of Overfitting and Generalization in Continuous Reinforcement Learning
• How Many Random Seeds? Statistical Power Analysis in Deep Reinforcement Learning Experiments

Cited by in corpus (16)

• A Survey of Zero-shot Generalisation in Deep Reinforcement Learning
• Wasserstein Robust Reinforcement Learning
• Analysing Deep Reinforcement Learning Agents Trained with Domain Randomisation
• Generalization in Transfer Learning
• Regularization Matters in Policy Optimization
• Evolving and Merging Hebbian Learning Rules: Increasing Generalization by Decreasing the Number of Rules
• Transient Non-Stationarity and Generalisation in Deep Reinforcement Learning
• Improving Generalization of Reinforcement Learning with Minimax Distributional Soft Actor-Critic
• Measuring Visual Generalization in Continuous Control from Pixels
• Robust Domain Randomised Reinforcement Learning through Peer-to-Peer Distillation
• Off-Policy Meta-Reinforcement Learning Based on Feature Embedding Spaces
• LISPR: An Options Framework for Policy Reuse with Reinforcement Learning
• Efficient decorrelation of features using Gramian in Reinforcement Learning
• CoachNet: An Adversarial Sampling Approach for Reinforcement Learning
• Learning Transferable Concepts in Deep Reinforcement Learning
• Learning robust driving policies without online exploration