Sequence Modeling of Temporal Credit Assignment for Episodic Reinforcement Learning

Abstract

Recent advances in deep reinforcement learning algorithms have shown great potential and success for solving many challenging real-world problems, including Go game and robotic applications. Usually, these algorithms need a carefully designed reward function to guide training in each time step. However, in real world, it is non-trivial to design such a reward function, and the only signal available is usually obtained at the end of a trajectory, also known as the episodic reward or return. In this work, we introduce a new algorithm for temporal credit assignment, which learns to decompose the episodic return back to each time-step in the trajectory using deep neural networks. With this learned reward signal, the learning efficiency can be substantially improved for episodic reinforcement learning. In particular, we find that expressive language models such as the Transformer can be adopted for learning the importance and the dependency of states in the trajectory, therefore providing high-quality and interpretable learned reward signals. We have performed extensive experiments on a set of MuJoCo continuous locomotive control tasks with only episodic returns and demonstrated the effectiveness of our algorithm.

References in corpus (7)

• A Structured Self-attentive Sentence Embedding
• Evolution Strategies as a Scalable Alternative to Reinforcement Learning
• Emergence of Locomotion Behaviours in Rich Environments
• Incentivizing Exploration In Reinforcement Learning With Deep Predictive Models
• Evolved Policy Gradients
• Generative Adversarial Self-Imitation Learning
• On Learning Intrinsic Rewards for Policy Gradient Methods

Cited by in corpus (6)

• Decision Transformer: Reinforcement Learning via Sequence Modeling
• Agent57: Outperforming the Atari Human Benchmark
• Off-Policy Reinforcement Learning with Delayed Rewards
• Synthetic Returns for Long-Term Credit Assignment
• Learning Long-Term Reward Redistribution via Randomized Return Decomposition
• Learning Guidance Rewards with Trajectory-space Smoothing