CAQL: Continuous Action Q-Learning

Abstract

Value-based reinforcement learning (RL) methods like Q-learning have shown success in a variety of domains. One challenge in applying Q-learning to continuous-action RL problems, however, is the continuous action maximization (max-Q) required for optimal Bellman backup. In this work, we develop CAQL, a (class of) algorithm(s) for continuous-action Q-learning that can use several plug-and-play optimizers for the max-Q problem. Leveraging recent optimization results for deep neural networks, we show that max-Q can be solved optimally using mixed-integer programming (MIP). When the Q-function representation has sufficient power, MIP-based optimization gives rise to better policies and is more robust than approximate methods (e.g., gradient ascent, cross-entropy search). We further develop several techniques to accelerate inference in CAQL, which despite their approximate nature, perform well. We compare CAQL with state-of-the-art RL algorithms on benchmark continuous-control problems that have different degrees of action constraints and show that CAQL outperforms policy-based methods in heavily constrained environments, often dramatically.

References in corpus (14)

• Playing Atari with Deep Reinforcement Learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation
• Evaluating Robustness of Neural Networks with Mixed Integer Programming
• Continuous Deep Q-Learning with Model-based Acceleration
• Global Convergence of Policy Gradient Methods for the Linear Quadratic Regulator
• Challenges of Real-World Reinforcement Learning
• An approach to reachability analysis for feed-forward ReLU neural networks
• Input Convex Neural Networks
• Towards Fast Computation of Certified Robustness for ReLU Networks
• A Unified View of Piecewise Linear Neural Network Verification
• A unified view of entropy-regularized Markov decision processes
• Horizon: Facebook's Open Source Applied Reinforcement Learning Platform
• Equivalent and Approximate Transformations of Deep Neural Networks

Cited by in corpus (4)

• A Q-learning approach to the continuous control problem of robot inverted pendulum balancing
• Hamilton-Jacobi Deep Q-Learning for Deterministic Continuous-Time Systems with Lipschitz Continuous Controls
• Constrained Model-Free Reinforcement Learning for Process Optimization
• Importance Sampling based Exploration in Q Learning