REPAINT: Knowledge Transfer in Deep Reinforcement Learning

Abstract

Accelerating learning processes for complex tasks by leveraging previously learned tasks has been one of the most challenging problems in reinforcement learning, especially when the similarity between source and target tasks is low. This work proposes REPresentation And INstance Transfer (REPAINT) algorithm for knowledge transfer in deep reinforcement learning. REPAINT not only transfers the representation of a pre-trained teacher policy in the on-policy learning, but also uses an advantage-based experience selection approach to transfer useful samples collected following the teacher policy in the off-policy learning. Our experimental results on several benchmark tasks show that REPAINT significantly reduces the total training time in generic cases of task similarity. In particular, when the source tasks are dissimilar to, or sub-tasks of, the target tasks, REPAINT outperforms other baselines in both training-time reduction and asymptotic performance of return scores.

arXiv commentPublished at ICML 2021

References in corpus (19)

• Trust Region Policy Optimization
• Prioritized Experience Replay
• High-Dimensional Continuous Control Using Generalized Advantage Estimation
• Dota 2 with Large Scale Deep Reinforcement Learning
• StarCraft II: A New Challenge for Reinforcement Learning
• RL${}^2$: Fast Reinforcement Learning via Slow Reinforcement Learning
• Actor-Mimic: Deep Multitask and Transfer Reinforcement Learning
• Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning
• Critic Regularized Regression
• Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning
• Kickstarting Deep Reinforcement Learning
• Transfer in Deep Reinforcement Learning Using Successor Features and Generalised Policy Improvement
• Decoupling Dynamics and Reward for Transfer Learning
• DeepRacer: Educational Autonomous Racing Platform for Experimentation with Sim2Real Reinforcement Learning
• Importance Weighted Transfer of Samples in Reinforcement Learning
• Universal Successor Features Approximators
• Better transfer learning with inferred successor maps
• Universal Successor Representations for Transfer Reinforcement Learning
• Generalized Off-Policy Actor-Critic

Cited by in corpus (1)

• Lifelong Robotic Reinforcement Learning by Retaining Experiences