Regularization Matters in Policy Optimization

Abstract

Deep Reinforcement Learning (Deep RL) has been receiving increasingly more attention thanks to its encouraging performance on a variety of control tasks. Yet, conventional regularization techniques in training neural networks (e.g., $L_2$ regularization, dropout) have been largely ignored in RL methods, possibly because agents are typically trained and evaluated in the same environment, and because the deep RL community focuses more on high-level algorithm designs. In this work, we present the first comprehensive study of regularization techniques with multiple policy optimization algorithms on continuous control tasks. Interestingly, we find conventional regularization techniques on the policy networks can often bring large improvement, especially on harder tasks. Our findings are shown to be robust against training hyperparameter variations. We also compare these techniques with the more widely used entropy regularization. In addition, we study regularizing different components and find that only regularizing the policy network is typically the best. We further analyze why regularization may help generalization in RL from four perspectives - sample complexity, reward distribution, weight norm, and noise robustness. We hope our study provides guidance for future practices in regularizing policy optimization algorithms. Our code is available at https://github.com/xuanlinli17/iclr2021_rlreg .

arXiv commentPublished at ICLR 2021; please cite this paper's ICLR 2021 version at https://github.com/xuanlinli17/iclr2021_rlreg#citation or the arXiv version from "Export Bibtex Citation" on the right, instead of the "2019 OpenReview" version in Google Scholar. Thanks!

References in corpus (20)

• Continuous control with deep reinforcement learning
• Decoupled Weight Decay Regularization
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Trust Region Policy Optimization
• Addressing Function Approximation Error in Actor-Critic Methods
• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
• RL${}^2$: Fast Reinforcement Learning via Slow Reinforcement Learning
• Deep Reinforcement Learning that Matters
• Reinforcement Learning with Augmented Data
• A Study on Overfitting in Deep Reinforcement Learning
• Quantifying Generalization in Reinforcement Learning
• Meta-Reinforcement Learning of Structured Exploration Strategies
• Continuous Adaptation via Meta-Learning in Nonstationary and Competitive Environments
• Assessing Generalization in Deep Reinforcement Learning
• A unified view of entropy-regularized Markov decision processes
• Generalization and Regularization in DQN
• A Dissection of Overfitting and Generalization in Continuous Reinforcement Learning
• Control Regularization for Reduced Variance Reinforcement Learning
• Investigating Generalisation in Continuous Deep Reinforcement Learning

Cited by in corpus (8)

• Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels
• Review: Deep Learning in Electron Microscopy
• IPAPRec: A promising tool for learning high-performance mapless navigation skills with deep reinforcement learning
• F2A2: Flexible Fully-decentralized Approximate Actor-critic for Cooperative Multi-agent Reinforcement Learning
• Training Larger Networks for Deep Reinforcement Learning
• Transient Non-Stationarity and Generalisation in Deep Reinforcement Learning
• A Hybrid Stochastic Policy Gradient Algorithm for Reinforcement Learning
• Proximal Policy Gradient: PPO with Policy Gradient