An Information-Theoretic Perspective on Credit Assignment in Reinforcement Learning
arXiv:2103.06224
Abstract
How do we formalize the challenge of credit assignment in reinforcement learning? Common intuition would draw attention to reward sparsity as a key contributor to difficult credit assignment and traditional heuristics would look to temporal recency for the solution, calling upon the classic eligibility trace. We posit that it is not the sparsity of the reward itself that causes difficulty in credit assignment, but rather the \emph{information sparsity}. We propose to use information theory to define this notion, which we then use to characterize when credit assignment is an obstacle to efficient learning. With this perspective, we outline several information-theoretic mechanisms for measuring credit under a fixed behavior policy, highlighting the potential of information theory as a key tool towards provably-efficient credit assignment.
arXiv commentWorkshop on Biological and Artificial Reinforcement Learning (NeurIPS 2020)
References in corpus (15)
- Rainbow: Combining Improvements in Deep Reinforcement Learning
- Reinforcement Learning with Deep Energy-Based Policies
- Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review
- Optimal control as a graphical model inference problem
- Thermodynamics as a theory of decision-making with information processing costs
- Deep Exploration via Randomized Value Functions
- The relation between Granger causality and directed information theory: a review
- Parametric Return Density Estimation for Reinforcement Learning
- EMI: Exploration with Mutual Information
- A Unified Bellman Equation for Causal Information and Value in Markov Decision Processes
- Keeping Your Distance: Solving Sparse Reward Tasks Using Self-Balancing Shaped Rewards
- Exploiting Hierarchy for Learning and Transfer in KL-regularized RL
- Hindsight Credit Assignment
- Self-Attentional Credit Assignment for Transfer in Reinforcement Learning
- Reinforcement Learning Evaluation and Solution for the Feedback Capacity of the Ising Channel with Large Alphabet