Synthetic Returns for Long-Term Credit Assignment

Abstract

Since the earliest days of reinforcement learning, the workhorse method for assigning credit to actions over time has been temporal-difference (TD) learning, which propagates credit backward timestep-by-timestep. This approach suffers when delays between actions and rewards are long and when intervening unrelated events contribute variance to long-term returns. We propose state-associative (SA) learning, where the agent learns associations between states and arbitrarily distant future rewards, then propagates credit directly between the two. In this work, we use SA-learning to model the contribution of past states to the current reward. With this model we can predict each state's contribution to the far future, a quantity we call "synthetic returns". TD-learning can then be applied to select actions that maximize these synthetic returns (SRs). We demonstrate the effectiveness of augmenting agents with SRs across a range of tasks on which TD-learning alone fails. We show that the learned SRs are interpretable: they spike for states that occur after critical actions are taken. Finally, we show that our IMPALA-based SR agent solves Atari Skiing -- a game with a lengthy reward delay that posed a major hurdle to deep-RL agents -- 25 times faster than the published state-of-the-art.

References in corpus (12)

• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
• Distributed Prioritized Experience Replay
• Rainbow: Combining Improvements in Deep Reinforcement Learning
• Agent57: Outperforming the Atari Human Benchmark
• Control of Memory, Active Perception, and Action in Minecraft
• Unsupervised Predictive Memory in a Goal-Directed Agent
• Woulda, Coulda, Shoulda: Counterfactually-Guided Policy Search
• Been There, Done That: Meta-Learning with Episodic Recall
• Generalization of Reinforcement Learners with Working and Episodic Memory
• Sequence Modeling of Temporal Credit Assignment for Episodic Reinforcement Learning
• Self-Attentional Credit Assignment for Transfer in Reinforcement Learning
• Counterfactual Credit Assignment in Model-Free Reinforcement Learning

Cited by in corpus (4)

• Decision Transformer: Reinforcement Learning via Sequence Modeling
• Deep Reinforcement Learning at the Edge of the Statistical Precipice
• Learning Long-Term Reward Redistribution via Randomized Return Decomposition
• Towards Practical Credit Assignment for Deep Reinforcement Learning