Deterministic Implementations for Reproducibility in Deep Reinforcement Learning

Abstract

While deep reinforcement learning (DRL) has led to numerous successes in recent years, reproducing these successes can be extremely challenging. One reproducibility challenge particularly relevant to DRL is nondeterminism in the training process, which can substantially affect the results. Motivated by this challenge, we study the positive impacts of deterministic implementations in eliminating nondeterminism in training. To do so, we consider the particular case of the deep Q-learning algorithm, for which we produce a deterministic implementation by identifying and controlling all sources of nondeterminism in the training process. One by one, we then allow individual sources of nondeterminism to affect our otherwise deterministic implementation, and measure the impact of each source on the variance in performance. We find that individual sources of nondeterminism can substantially impact the performance of agent, illustrating the benefits of deterministic implementations. In addition, we also discuss the important role of deterministic implementations in achieving exact replicability of results.

arXiv comment17 Pages

References in corpus (6)

• Playing Atari with Deep Reinforcement Learning
• Emergence of Locomotion Behaviours in Rich Environments
• DeepMind Control Suite
• Massively Parallel Methods for Deep Reinforcement Learning
• Reproducibility of Benchmarked Deep Reinforcement Learning Tasks for Continuous Control
• How Many Random Seeds? Statistical Power Analysis in Deep Reinforcement Learning Experiments

Cited by in corpus (11)

• A Survey and Critique of Multiagent Deep Reinforcement Learning
• Deep Reinforcement Learning at the Edge of the Statistical Precipice
• Measuring the Reliability of Reinforcement Learning Algorithms
• The Scientific Method in the Science of Machine Learning
• Smooth activations and reproducibility in deep networks
• Anti-Distillation: Improving reproducibility of deep networks
• Catalyst.RL: A Distributed Framework for Reproducible RL Research
• Generalization Guarantees for Neural Architecture Search with Train-Validation Split
• A Study of Checkpointing in Large Scale Training of Deep Neural Networks
• Synthesizing Irreproducibility in Deep Networks
• A Comparison of the Delta Method and the Bootstrap in Deep Learning Classification