Counterfactual Credit Assignment in Model-Free Reinforcement Learning
arXiv:2011.09464
Abstract
Credit assignment in reinforcement learning is the problem of measuring an action's influence on future rewards. In particular, this requires separating skill from luck, i.e. disentangling the effect of an action on rewards from that of external factors and subsequent actions. To achieve this, we adapt the notion of counterfactuals from causality theory to a model-free RL setup. The key idea is to condition value functions on future events, by learning to extract relevant information from a trajectory. We formulate a family of policy gradient algorithms that use these future-conditional value functions as baselines or critics, and show that they are provably low variance. To avoid the potential bias from conditioning on future information, we constrain the hindsight information to not contain information about the agent's actions. We demonstrate the efficacy and validity of our algorithm on a number of illustrative and challenging problems.
References in corpus (16)
- Adam: A Method for Stochastic Optimization
- Trust Region Policy Optimization
- Counterfactual Multi-Agent Policy Gradients
- Gated Feedback Recurrent Neural Networks
- Model-Based Reinforcement Learning for Atari
- Learning Continuous Control Policies by Stochastic Value Gradients
- Benchmarking Model-Based Reinforcement Learning
- The Optimal Reward Baseline for Gradient-Based Reinforcement Learning
- Stabilizing Transformers for Reinforcement Learning
- RUDDER: Return Decomposition for Delayed Rewards
- Variance Reduction for Policy Gradient with Action-Dependent Factorized Baselines
- Recurrent Independent Mechanisms
- Hindsight policy gradients
- Hindsight Credit Assignment
- Credit Assignment Techniques in Stochastic Computation Graphs
- Variance Reduced Advantage Estimation with Hindsight Credit Assignment