Control Regularization for Reduced Variance Reinforcement Learning
arXiv:1905.05380
Abstract
Dealing with high variance is a significant challenge in model-free reinforcement learning (RL). Existing methods are unreliable, exhibiting high variance in performance from run to run using different initializations/seeds. Focusing on problems arising in continuous control, we propose a functional regularization approach to augmenting model-free RL. In particular, we regularize the behavior of the deep policy to be similar to a policy prior, i.e., we regularize in function space. We show that functional regularization yields a bias-variance trade-off, and propose an adaptive tuning strategy to optimize this trade-off. When the policy prior has control-theoretic stability guarantees, we further show that this regularization approximately preserves those stability guarantees throughout learning. We validate our approach empirically on a range of settings, and demonstrate significantly reduced variance, guaranteed dynamic stability, and more efficient learning than deep RL alone.
arXiv commentAppearing in ICML 2019
Cited by in corpus (13)
- Contraction Theory for Nonlinear Stability Analysis and Learning-based Control: A Tutorial Overview
- The Stability-Efficiency Dilemma: Investigating Sequence Length Warmup for Training GPT Models
- Regularization Matters in Policy Optimization
- Imitation-Projected Programmatic Reinforcement Learning
- Robust Regression for Safe Exploration in Control
- Neural-Swarm: Decentralized Close-Proximity Multirotor Control Using Learned Interactions
- Neural-Swarm2: Planning and Control of Heterogeneous Multirotor Swarms using Learned Interactions
- A3C-S: Automated Agent Accelerator Co-Search towards Efficient Deep Reinforcement Learning
- Stabilizing Neural Control Using Self-Learned Almost Lyapunov Critics
- Learning Deep Energy Shaping Policies for Stability-Guaranteed Manipulation
- Adaptive Experience Selection for Policy Gradient
- Stochastic Intervention for Causal Inference via Reinforcement Learning
- Coordinate-wise Control Variates for Deep Policy Gradients