Relative Entropy Regularized Policy Iteration

Abstract

We present an off-policy actor-critic algorithm for Reinforcement Learning (RL) that combines ideas from gradient-free optimization via stochastic search with learned action-value function. The result is a simple procedure consisting of three steps: i) policy evaluation by estimating a parametric action-value function; ii) policy improvement via the estimation of a local non-parametric policy; and iii) generalization by fitting a parametric policy. Each step can be implemented in different ways, giving rise to several algorithm variants. Our algorithm draws on connections to existing literature on black-box optimization and 'RL as an inference' and it can be seen either as an extension of the Maximum a Posteriori Policy Optimisation algorithm (MPO) [Abdolmaleki et al., 2018a], or as an extension of Trust Region Covariance Matrix Adaptation Evolutionary Strategy (CMA-ES) [Abdolmaleki et al., 2017b; Hansen et al., 1997] to a policy iteration scheme. Our comparison on 31 continuous control tasks from parkour suite [Heess et al., 2017], DeepMind control suite [Tassa et al., 2018] and OpenAI Gym [Brockman et al., 2016] with diverse properties, limited amount of compute and a single set of hyperparameters, demonstrate the effectiveness of our method and the state of art results. Videos, summarizing results, can be found at goo.gl/HtvJKR .

References in corpus (14)

• Adam: A Method for Stochastic Optimization
• Continuous control with deep reinforcement learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Benchmarking Deep Reinforcement Learning for Continuous Control
• Emergence of Locomotion Behaviours in Rich Environments
• DeepMind Control Suite
• Learning Continuous Control Policies by Stochastic Value Gradients
• Learning by Playing - Solving Sparse Reward Tasks from Scratch
• Maximum a Posteriori Policy Optimisation
• Bridging the Gap Between Value and Policy Based Reinforcement Learning
• Emergent Complexity via Multi-Agent Competition
• Learning and Transfer of Modulated Locomotor Controllers
• Path Integral Guided Policy Search
• Guide Actor-Critic for Continuous Control

Cited by in corpus (20)

• Recent advances in applying deep reinforcement learning for flow control: perspectives and future directions
• Stabilizing Transformers for Reinforcement Learning
• Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Model
• Robust Reinforcement Learning for Continuous Control with Model Misspecification
• Discrete and Continuous Action Representation for Practical RL in Video Games
• V-MPO: On-Policy Maximum a Posteriori Policy Optimization for Discrete and Continuous Control
• Continuous-Discrete Reinforcement Learning for Hybrid Control in Robotics
• Q-Learning for Continuous Actions with Cross-Entropy Guided Policies
• Learning and Planning in Complex Action Spaces
• Data-efficient Hindsight Off-policy Option Learning
• TAAC: Temporally Abstract Actor-Critic for Continuous Control
• Imagined Value Gradients: Model-Based Policy Optimization with Transferable Latent Dynamics Models
• Robust Constrained Reinforcement Learning for Continuous Control with Model Misspecification
• Policy Search by Target Distribution Learning for Continuous Control
• Is Bang-Bang Control All You Need? Solving Continuous Control with Bernoulli Policies
• Zeroth-Order Supervised Policy Improvement
• A Relation Analysis of Markov Decision Process Frameworks
• Measuring Progress in Deep Reinforcement Learning Sample Efficiency
• Bayes-Adaptive Deep Model-Based Policy Optimisation
• Co-Adaptation of Algorithmic and Implementational Innovations in Inference-based Deep Reinforcement Learning