An Information-Theoretic Perspective on Credit Assignment in Reinforcement Learning
arXiv:2103.06224
Abstract
How do we formalize the challenge of credit assignment in reinforcement learning? Common intuition would draw attention to reward sparsity as a key contributor to difficult credit assignment and traditional heuristics would look to temporal recency for the solution, calling upon the classic eligibility trace. We posit that it is not the sparsity of the reward itself that causes difficulty in credit assignment, but rather the \emph{information sparsity}. We propose to use information theory to define this notion, which we then use to characterize when credit assignment is an obstacle to efficient learning. With this perspective, we outline several information-theoretic mechanisms for measuring credit under a fixed behavior policy, highlighting the potential of information theory as a key tool towards provably-efficient credit assignment.
Workshop on Biological and Artificial Reinforcement Learning (NeurIPS 2020)
References in corpus (5)
- Rainbow: Combining Improvements in Deep Reinforcement Learning
- Parametric Return Density Estimation for Reinforcement Learning
- Keeping Your Distance: Solving Sparse Reward Tasks Using Self-Balancing Shaped Rewards
- Hindsight Credit Assignment
- Reinforcement Learning Evaluation and Solution for the Feedback Capacity of the Ising Channel with Large Alphabet