Spectral Normalisation for Deep Reinforcement Learning: an Optimisation Perspective
arXiv:2105.05246
Abstract
Most of the recent deep reinforcement learning advances take an RL-centric perspective and focus on refinements of the training objective. We diverge from this view and show we can recover the performance of these developments not by changing the objective, but by regularising the value-function estimator. Constraining the Lipschitz constant of a single layer using spectral normalisation is sufficient to elevate the performance of a Categorical-DQN agent to that of a more elaborated \rainbow{} agent on the challenging Atari domain. We conduct ablation studies to disentangle the various effects normalisation has on the learning dynamics and show that is sufficient to modulate the parameter updates to recover most of the performance of spectral normalisation. These findings hint towards the need to also focus on the neural component and its learning dynamics to tackle the peculiarities of Deep Reinforcement Learning.
arXiv commentAccepted at ICML2021
References in corpus (24)
- Adam: A Method for Stochastic Optimization
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- Spectral Normalization for Generative Adversarial Networks
- Dueling Network Architectures for Deep Reinforcement Learning
- Noisy Networks for Exploration
- Optimizing Neural Networks with Kronecker-factored Approximate Curvature
- Rainbow: Combining Improvements in Deep Reinforcement Learning
- Spectral Norm Regularization for Improving the Generalizability of Deep Learning
- L2 Regularization versus Batch and Weight Normalization
- Dopamine: A Research Framework for Deep Reinforcement Learning
- Parseval Networks: Improving Robustness to Adversarial Examples
- Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awareness
- Generalizable Adversarial Training via Spectral Normalization
- Theoretical Analysis of Auto Rate-Tuning by Batch Normalization
- MinAtar: An Atari-Inspired Testbed for Thorough and Reproducible Reinforcement Learning Experiments
- Revisiting Rainbow: Promoting more Insightful and Inclusive Deep Reinforcement Learning Research
- Is Deep Reinforcement Learning Really Superhuman on Atari? Leveling the playing field
- A case for new neural network smoothness constraints
- Where Did My Optimum Go?: An Empirical Analysis of Gradient Descent Optimization in Policy Gradient Methods
- Spectral Normalisation for Deep Reinforcement Learning: an Optimisation Perspective
- Interference and Generalization in Temporal Difference Learning
- Adaptive Temporal Difference Learning with Linear Function Approximation
- Implicit Under-Parameterization Inhibits Data-Efficient Deep Reinforcement Learning
- TDprop: Does Jacobi Preconditioning Help Temporal Difference Learning?