How to Make Deep RL Work in Practice
arXiv:2010.13083
Abstract
In recent years, challenging control problems became solvable with deep reinforcement learning (RL). To be able to use RL for large-scale real-world applications, a certain degree of reliability in their performance is necessary. Reported results of state-of-the-art algorithms are often difficult to reproduce. One reason for this is that certain implementation details influence the performance significantly. Commonly, these details are not highlighted as important techniques to achieve state-of-the-art performance. Additionally, techniques from supervised learning are often used by default but influence the algorithms in a reinforcement learning setting in different and not well-understood ways. In this paper, we investigate the influence of certain initialization, input normalization, and adaptive learning techniques on the performance of state-of-the-art RL algorithms. We make suggestions which of those techniques to use by default and highlight areas that could benefit from a solution specifically tailored to RL.
arXiv commentPublished as a workshop paper at the Deep RL workshop, NeurIPS 2020
References in corpus (12)
- Playing Atari with Deep Reinforcement Learning
- Addressing Function Approximation Error in Actor-Critic Methods
- Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
- Benchmarking Deep Reinforcement Learning for Continuous Control
- QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation
- Solving Rubik's Cube with a Robot Hand
- DeepMind Control Suite
- Reproducibility of Benchmarked Deep Reinforcement Learning Tasks for Continuous Control
- How Many Random Seeds? Statistical Power Analysis in Deep Reinforcement Learning Experiments
- The Mirage of Action-Dependent Baselines in Reinforcement Learning
- Q-Learning for Continuous Actions with Cross-Entropy Guided Policies
- Time Limits in Reinforcement Learning