Self-Attentional Credit Assignment for Transfer in Reinforcement Learning

Abstract

The ability to transfer knowledge to novel environments and tasks is a sensible desiderata for general learning agents. Despite the apparent promises, transfer in RL is still an open and little exploited research area. In this paper, we take a brand-new perspective about transfer: we suggest that the ability to assign credit unveils structural invariants in the tasks that can be transferred to make RL more sample-efficient. Our main contribution is SECRET, a novel approach to transfer learning for RL that uses a backward-view credit assignment mechanism based on a self-attentive architecture. Two aspects are key to its generality: it learns to assign credit as a separate offline supervised process and exclusively modifies the reward function. Consequently, it can be supplemented by transfer methods that do not modify the reward function and it can be plugged on top of any RL algorithm.

arXiv comment21 pages, 10 figures, 3 tables (accepted as an oral presentation at the Learning Transferable Skills workshop, NeurIPS 2019)

References in corpus (24)

• Neural Machine Translation by Jointly Learning to Align and Translate
• Sequence to Sequence Learning with Neural Networks
• Overcoming catastrophic forgetting in neural networks
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• High-Dimensional Continuous Control Using Generalized Advantage Estimation
• A Structured Self-attentive Sentence Embedding
• Progressive Neural Networks
• A Simple Neural Attentive Meta-Learner
• Generating Long Sequences with Sparse Transformers
• Learning to reinforcement learn
• Go-Explore: a New Approach for Hard-Exploration Problems
• Episodic Curiosity through Reachability
• Distral: Robust Multitask Reinforcement Learning
• Evolved Policy Gradients
• DARLA: Improving Zero-Shot Transfer in Reinforcement Learning
• Assessing Generalization in Deep Reinforcement Learning
• RUDDER: Return Decomposition for Delayed Rewards
• Policy Distillation
• Learning to Generalize from Sparse and Underspecified Rewards
• Reward Shaping via Meta-Learning
• Kickstarting Deep Reinforcement Learning
• DeepMind Lab
• Sparse Attentive Backtracking: Temporal CreditAssignment Through Reminding
• Multi-task Deep Reinforcement Learning with PopArt

Cited by in corpus (5)

• Decision Transformer: Reinforcement Learning via Sequence Modeling
• Agent57: Outperforming the Atari Human Benchmark
• Synthetic Returns for Long-Term Credit Assignment
• Adversarially Guided Actor-Critic
• An Information-Theoretic Perspective on Credit Assignment in Reinforcement Learning