A Deeper Look at Discounting Mismatch in Actor-Critic Algorithms

Abstract

We investigate the discounting mismatch in actor-critic algorithm implementations from a representation learning perspective. Theoretically, actor-critic algorithms usually have discounting for both actor and critic, i.e., there is a ${\gamma }^t$ term in the actor update for the transition observed at time $t$ in a trajectory and the critic is a discounted value function. Practitioners, however, usually ignore the discounting (${\gamma }^t$) for the actor while using a discounted critic. We investigate this mismatch in two scenarios. In the first scenario, we consider optimizing an undiscounted objective $(\gamma =1)$ where ${\gamma }^t$ disappears naturally $(1^t=1)$. We then propose to interpret the discounting in critic in terms of a bias-variance-representation trade-off and provide supporting empirical results. In the second scenario, we consider optimizing a discounted objective ($\gamma <1$) and propose to interpret the omission of the discounting in the actor update from an auxiliary task perspective and provide supporting empirical results.

arXiv commentAAMAS 2022

References in corpus (16)

• Adam: A Method for Stochastic Optimization
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Trust Region Policy Optimization
• Addressing Function Approximation Error in Actor-Critic Methods
• High-Dimensional Continuous Control Using Generalized Advantage Estimation
• On the Theory of Policy Gradient Methods: Optimality, Approximation, and Distribution Shift
• What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Study
• Hyperbolic Discounting and Learning over Multiple Horizons
• rlpyt: A Research Code Base for Deep Reinforcement Learning in PyTorch
• Sample Efficient Policy Gradient Methods with Recursive Variance Reduction
• Discovery of Useful Questions as Auxiliary Tasks
• A Finite Time Analysis of Two Time-Scale Actor Critic Methods
• A Closer Look at Deep Policy Gradients
• Discount Factor as a Regularizer in Reinforcement Learning
• Stochastic Recursive Momentum for Policy Gradient Methods
• Multi-Advisor Reinforcement Learning

Cited by in corpus (1)

• Dr Jekyll and Mr Hyde: the Strange Case of Off-Policy Policy Updates