Finding the Near Optimal Policy via Adaptive Reduced Regularization in MDPs
arXiv:2011.00213
Abstract
Regularized MDPs serve as a smooth version of original MDPs. However, biased optimal policy always exists for regularized MDPs. Instead of making the coefficientλof regularized term sufficiently small, we propose an adaptive reduction scheme for λ to approximate optimal policy of the original MDP. It is shown that the iteration complexity for obtaining anε-optimal policy could be reduced in comparison with setting sufficiently smallλ. In addition, there exists strong duality connection between the reduction method and solving the original MDP directly, from which we can derive more adaptive reduction method for certain algorithms.
References in corpus (10)
- Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
- Trust Region Policy Optimization
- Equivalence Between Policy Gradients and Soft Q-Learning
- SBEED: Convergent Reinforcement Learning with Nonlinear Function Approximation
- On the Theory of Policy Gradient Methods: Optimality, Approximation, and Distribution Shift
- A unified view of entropy-regularized Markov decision processes
- Neural Policy Gradient Methods: Global Optimality and Rates of Convergence
- Taming the Noise in Reinforcement Learning via Soft Updates
- Fast Global Convergence of Natural Policy Gradient Methods with Entropy Regularization
- On the Convergence of Approximate and Regularized Policy Iteration Schemes