Logistic Q-Learning
arXiv:2010.11151
Abstract
We propose a new reinforcement learning algorithm derived from a regularized linear-programming formulation of optimal control in MDPs. The method is closely related to the classic Relative Entropy Policy Search (REPS) algorithm of Peters et al. (2010), with the key difference that our method introduces a Q-function that enables efficient exact model-free implementation. The main feature of our algorithm (called QREPS) is a convex loss function for policy evaluation that serves as a theoretically sound alternative to the widely used squared Bellman error. We provide a practical saddle-point optimization method for minimizing this loss function and provide an error-propagation analysis that relates the quality of the individual updates to the performance of the output policy. Finally, we demonstrate the effectiveness of our method on a range of benchmark problems.
References in corpus (16)
- Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
- Trust Region Policy Optimization
- Addressing Function Approximation Error in Actor-Critic Methods
- Maximum a Posteriori Policy Optimisation
- Provably Efficient Reinforcement Learning with Linear Function Approximation
- SBEED: Convergent Reinforcement Learning with Nonlinear Function Approximation
- A Theoretical Analysis of Deep Q-Learning
- A unified view of entropy-regularized Markov decision processes
- A Theory of Regularized Markov Decision Processes
- Model-Based Reinforcement Learning with Value-Targeted Regression
- V-MPO: On-Policy Maximum a Posteriori Policy Optimization for Discrete and Continuous Control
- Munchausen Reinforcement Learning
- FLAMBE: Structural Complexity and Representation Learning of Low Rank MDPs
- A Kernel Loss for Solving the Bellman Equation
- Convex Q-Learning, Part 1: Deterministic Optimal Control
- A Unifying View of Optimism in Episodic Reinforcement Learning