Is High Variance Unavoidable in RL? A Case Study in Continuous Control

Abstract

Reinforcement learning (RL) experiments have notoriously high variance, and minor details can have disproportionately large effects on measured outcomes. This is problematic for creating reproducible research and also serves as an obstacle for real-world applications, where safety and predictability are paramount. In this paper, we investigate causes for this perceived instability. To allow for an in-depth analysis, we focus on a specifically popular setup with high variance -- continuous control from pixels with an actor-critic agent. In this setting, we demonstrate that variance mostly arises early in training as a result of poor "outlier" runs, but that weight initialization and initial exploration are not to blame. We show that one cause for early variance is numerical instability which leads to saturating nonlinearities. We investigate several fixes to this issue and find that one particular method is surprisingly effective and simple -- normalizing penultimate features. Addressing the learning instability allows for larger learning rates, and significantly decreases the variance of outcomes. This demonstrates that the perceived variance in RL is not necessarily inherent to the problem definition and may be addressed through simple architectural modifications.

arXiv commentAccepted to ICLR2022

References in corpus (28)

• Continuous control with deep reinforcement learning
• A Simple Framework for Contrastive Learning of Visual Representations
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Bootstrap your own latent: A new approach to self-supervised Learning
• Convolutional Sequence to Sequence Learning
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Momentum Contrast for Unsupervised Visual Representation Learning
• Learning Representations by Maximizing Mutual Information Across Views
• Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction
• Reinforcement Learning with Augmented Data
• Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPO
• Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels
• Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networks
• Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learning
• Variance Reduction for Reinforcement Learning in Input-Driven Environments
• How Many Random Seeds? Statistical Power Analysis in Deep Reinforcement Learning Experiments
• Data-Efficient Reinforcement Learning with Self-Predictive Representations
• Measuring the Reliability of Reinforcement Learning Algorithms
• Understanding Batch Normalization
• A Hitchhiker's Guide to Statistical Comparisons of Reinforcement Learning Algorithms
• Learning to be Safe: Deep RL with a Safety Critic
• Meta-Learning MCMC Proposals
• On the adequacy of untuned warmup for adaptive optimization
• Let's Play Again: Variability of Deep Reinforcement Learning Agents in Atari Environments
• Learning Visual Robotic Control Efficiently with Contrastive Pre-training and Data Augmentation
• Significant Improvements over the State of the Art? A Case Study of the MS MARCO Document Ranking Leaderboard
• Variance Reduction for Deep Q-Learning using Stochastic Recursive Gradient
• A Survey on Reproducibility by Evaluating Deep Reinforcement Learning Algorithms on Real-World Robots