Local Search for Policy Iteration in Continuous Control

Abstract

We present an algorithm for local, regularized, policy improvement in reinforcement learning (RL) that allows us to formulate model-based and model-free variants in a single framework. Our algorithm can be interpreted as a natural extension of work on KL-regularized RL and introduces a form of tree search for continuous action spaces. We demonstrate that additional computation spent on model-based policy improvement during learning can improve data efficiency, and confirm that model-based policy improvement during action selection can also be beneficial. Quantitatively, our algorithm improves data efficiency on several continuous control benchmarks (when a model is learned in parallel), and it provides significant improvements in wall-clock time in high-dimensional domains (when a ground truth model is available). The unified framework also helps us to better understand the space of model-based and model-free algorithms. In particular, we demonstrate that some benefits attributed to model-based RL can be obtained without a model, simply by utilizing more computation.

References in corpus (25)

• Continuous control with deep reinforcement learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Trust Region Policy Optimization
• DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills
• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
• DeepMind Control Suite
• Learning Continuous Control Policies by Stochastic Value Gradients
• Learning Latent Dynamics for Planning from Pixels
• Path integrals and symmetry breaking for optimal control theory
• Maximum a Posteriori Policy Optimisation
• Thinking Fast and Slow with Deep Learning and Tree Search
• Equivalence Between Policy Gradients and Soft Q-Learning
• Dream to Control: Learning Behaviors by Latent Imagination
• Data-efficient Deep Reinforcement Learning for Dexterous Manipulation
• When to use parametric models in reinforcement learning?
• Hierarchical visuomotor control of humanoids
• Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Model
• V-MPO: On-Policy Maximum a Posteriori Policy Optimization for Discrete and Continuous Control
• A0C: Alpha Zero in Continuous Action Space
• Policy Gradient Search: Online Planning and Expert Iteration without Search Trees
• Guided Policy Search as Approximate Mirror Descent
• Particle Value Functions
• Imagined Value Gradients: Model-Based Policy Optimization with Transferable Latent Dynamics Models
• Approximate Inference in Discrete Distributions with Monte Carlo Tree Search and Value Functions
• Monte-Carlo Tree Search for Policy Optimization

Cited by in corpus (6)

• On the model-based stochastic value gradient for continuous reinforcement learning
• On the role of planning in model-based deep reinforcement learning
• Muesli: Combining Improvements in Policy Optimization
• Learning and Planning in Complex Action Spaces
• Entropy Regularized Motion Planning via Stein Variational Inference
• Evaluating model-based planning and planner amortization for continuous control