Explaining Off-Policy Actor-Critic From A Bias-Variance Perspective
arXiv:2110.02421
Abstract
Off-policy Actor-Critic algorithms have demonstrated phenomenal experimental performance but still require better explanations. To this end, we show its policy evaluation error on the distribution of transitions decomposes into: a Bellman error, a bias from policy mismatch, and a variance term from sampling. By comparing the magnitude of bias and variance, we explain the success of the Emphasizing Recent Experience sampling and 1/age weighted sampling. Both sampling strategies yield smaller bias and variance and are hence preferable to uniform sampling.
References in corpus (9)
- Trust Region Policy Optimization
- Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
- Provably Efficient Reinforcement Learning with Linear Function Approximation
- Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimation
- Boosting Soft Actor-Critic: Emphasizing Recent Experience without Forgetting the Past
- Near-Optimal Provable Uniform Convergence in Offline Policy Evaluation for Reinforcement Learning
- Experience Replay with Likelihood-free Importance Weights
- Learning to Combat Compounding-Error in Model-Based Reinforcement Learning
- Sample Complexity of Asynchronous Q-Learning: Sharper Analysis and Variance Reduction