A nearly Blackwell-optimal policy gradient method
arXiv:2105.13609
Abstract
For continuing environments, reinforcement learning (RL) methods commonly maximize the discounted reward criterion with discount factor close to 1 in order to approximate the average reward (the gain). However, such a criterion only considers the long-run steady-state performance, ignoring the transient behaviour in transient states. In this work, we develop a policy gradient method that optimizes the gain, then the bias (which indicates the transient performance and is important to capably select from policies with equal gain). We derive expressions that enable sampling for the gradient of the bias and its preconditioning Fisher matrix. We further devise an algorithm that solves the gain-then-bias (bi-level) optimization. Its key ingredient is an RL-specific logarithmic barrier function. Experimental results provide insights into the fundamental mechanisms of our proposal.
arXiv comment30 pages, major re-structuring
References in corpus (6)
- Meta-Gradient Reinforcement Learning
- An Improved Analysis of (Variance-Reduced) Policy Gradient and Natural Policy Gradient Methods
- Policy Evaluation Networks
- Average Reward Adjusted Discounted Reinforcement Learning: Near-Blackwell-Optimal Policies for Real-World Applications
- Zeroth-order Deterministic Policy Gradient
- Issues concerning realizability of Blackwell optimal policies in reinforcement learning