Biased Estimates of Advantages over Path Ensembles

Abstract

The estimation of advantage is crucial for a number of reinforcement learning algorithms, as it directly influences the choices of future paths. In this work, we propose a family of estimates based on the order statistics over the path ensemble, which allows one to flexibly drive the learning process, towards or against risks. On top of this formulation, we systematically study the impacts of different methods for estimating advantages. Our findings reveal that biased estimates, when chosen appropriately, can result in significant benefits. In particular, for the environments with sparse rewards, optimistic estimates would lead to more efficient exploration of the policy space; while for those where individual actions can have critical impacts, conservative estimates are preferable. On various benchmarks, including MuJoCo continuous control, Terrain locomotion, Atari games, and sparse-reward environments, the proposed biased estimation schemes consistently demonstrate improvement over mainstream methods, not only accelerating the learning process but also obtaining substantial performance gains.

References in corpus (12)

• Trust Region Policy Optimization
• Addressing Function Approximation Error in Actor-Critic Methods
• High-Dimensional Continuous Control Using Generalized Advantage Estimation
• StarCraft II: A New Challenge for Reinforcement Learning
• Emergence of Locomotion Behaviours in Rich Environments
• Distributed Distributional Deterministic Policy Gradients
• Implicit Quantile Networks for Distributional Reinforcement Learning
• Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approach
• Algorithms for CVaR Optimization in MDPs
• EX2: Exploration with Exemplar Models for Deep Reinforcement Learning
• Lightning Does Not Strike Twice: Robust MDPs with Coupled Uncertainty
• Distributionally Robust Reinforcement Learning