Towards Deeper Deep Reinforcement Learning with Spectral Normalization

Abstract

In computer vision and natural language processing, innovations in model architecture that increase model capacity have reliably translated into gains in performance. In stark contrast with this trend, state-of-the-art reinforcement learning (RL) algorithms often use small MLPs, and gains in performance typically originate from algorithmic innovations. It is natural to hypothesize that small datasets in RL necessitate simple models to avoid overfitting; however, this hypothesis is untested. In this paper we investigate how RL agents are affected by exchanging the small MLPs with larger modern networks with skip connections and normalization, focusing specifically on actor-critic algorithms. We empirically verify that naively adopting such architectures leads to instabilities and poor performance, likely contributing to the popularity of simple models in practice. However, we show that dataset size is not the limiting factor, and instead argue that instability from taking gradients through the critic is the culprit. We demonstrate that spectral normalization (SN) can mitigate this issue and enable stable training with large modern architectures. After smoothing with SN, larger models yield significant performance improvements -- suggesting that more "easy" gains may be had by focusing on model architectures in addition to algorithmic innovations.

arXiv commentaccepted NeurIPS 2021

References in corpus (32)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Continuous control with deep reinforcement learning
• On the difficulty of training Recurrent Neural Networks
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Spectral Normalization for Generative Adversarial Networks
• Dueling Network Architectures for Deep Reinforcement Learning
• Learning Latent Dynamics for Planning from Pixels
• MOPO: Model-based Offline Policy Optimization
• Reinforcement Learning with Augmented Data
• A Study on Overfitting in Deep Reinforcement Learning
• Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPO
• Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels
• On Layer Normalization in the Transformer Architecture
• Generalization and Regularization in DQN
• A Dissection of Overfitting and Generalization in Continuous Reinforcement Learning
• Learning Invariant Representations for Reinforcement Learning without Reconstruction
• Generalizable Adversarial Training via Spectral Normalization
• Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learning
• Data-Efficient Reinforcement Learning with Self-Predictive Representations
• SEED RL: Scalable and Efficient Deep-RL with Accelerated Central Inference
• Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learning
• Model-Based Offline Planning
• Return-Based Contrastive Representation Learning for Reinforcement Learning
• Sample Factory: Egocentric 3D Control from Pixels at 100000 FPS with Asynchronous Reinforcement Learning
• On the role of planning in model-based deep reinforcement learning
• D2RL: Deep Dense Architectures in Reinforcement Learning
• Spectral Normalisation for Deep Reinforcement Learning: an Optimisation Perspective
• High-Throughput Synchronous Deep RL
• Control-Aware Representations for Model-based Reinforcement Learning
• Causal Curiosity: RL Agents Discovering Self-supervised Experiments for Causal Representation Learning
• Offline Model-Based Optimization via Normalized Maximum Likelihood Estimation
• Self-supervised Visual Reinforcement Learning with Object-centric Representations