Return-based Scaling: Yet Another Normalisation Trick for Deep RL
arXiv:2105.05347
Abstract
Scaling issues are mundane yet irritating for practitioners of reinforcement learning. Error scales vary across domains, tasks, and stages of learning; sometimes by many orders of magnitude. This can be detrimental to learning speed and stability, create interference between learning tasks, and necessitate substantial tuning. We revisit this topic for agents based on temporal-difference learning, sketch out some desiderata and investigate scenarios where simple fixes fall short. The mechanism we propose requires neither tuning, clipping, nor adaptation. We validate its effectiveness and robustness on the suite of Atari games. Our scaling method turns out to be particularly helpful at mitigating interference, when training a shared neural network on multiple targets that differ in reward scale or discounting.
References in corpus (15)
- Adam: A Method for Stochastic Optimization
- Prioritized Experience Replay
- Dueling Network Architectures for Deep Reinforcement Learning
- IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
- Learning by Playing - Solving Sparse Reward Tasks from Scratch
- Agent57: Outperforming the Atari Human Benchmark
- Meta-Gradient Reinforcement Learning
- Generalization and Regularization in DQN
- Observe and Look Further: Achieving Consistent Performance on Atari
- Never Give Up: Learning Directed Exploration Strategies
- Decoupling Representation Learning from Reinforcement Learning
- Multi-task Deep Reinforcement Learning with PopArt
- General non-linear Bellman equations
- Adapting Behaviour for Learning Progress
- Adapting to Reward Progressivity via Spectral Reinforcement Learning