Q-Learning for Continuous Actions with Cross-Entropy Guided Policies

Abstract

Off-Policy reinforcement learning (RL) is an important class of methods for many problem domains, such as robotics, where the cost of collecting data is high and on-policy methods are consequently intractable. Standard methods for applying Q-learning to continuous-valued action domains involve iteratively sampling the Q-function to find a good action (e.g. via hill-climbing), or by learning a policy network at the same time as the Q-function (e.g. DDPG). Both approaches make tradeoffs between stability, speed, and accuracy. We propose a novel approach, called Cross-Entropy Guided Policies, or CGP, that draws inspiration from both classes of techniques. CGP aims to combine the stability and performance of iterative sampling policies with the low computational cost of a policy network. Our approach trains the Q-function using iterative sampling with the Cross-Entropy Method (CEM), while training a policy network to imitate CEM's sampling behavior. We demonstrate that our method is more stable to train than state of the art policy network methods, while preserving equivalent inference time compute costs, and achieving competitive total reward on standard benchmarks.

References in corpus (13)

• Playing Atari with Deep Reinforcement Learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Addressing Function Approximation Error in Actor-Critic Methods
• Evolution Strategies as a Scalable Alternative to Reinforcement Learning
• Benchmarking Deep Reinforcement Learning for Continuous Control
• Deep Reinforcement Learning that Matters
• Maximum a Posteriori Policy Optimisation
• Reproducibility of Benchmarked Deep Reinforcement Learning Tasks for Continuous Control
• Reinforcement and Imitation Learning for Diverse Visuomotor Skills
• Sim-to-Real Reinforcement Learning for Deformable Object Manipulation
• CEM-RL: Combining evolutionary and gradient-based methods for policy search
• Relative Entropy Regularized Policy Iteration
• Learning to Walk via Deep Reinforcement Learning

Cited by in corpus (6)

• Inverse Rational Control with Partially Observable Continuous Nonlinear Dynamics
• How to Make Deep RL Work in Practice
• GRAC: Self-Guided and Self-Regularized Actor-Critic
• Continuous Control with Action Quantization from Demonstrations
• Zeroth-Order Supervised Policy Improvement
• Importance Sampling based Exploration in Q Learning