Decoupling Dynamics and Reward for Transfer Learning

Abstract

Current reinforcement learning (RL) methods can successfully learn single tasks but often generalize poorly to modest perturbations in task domain or training procedure. In this work, we present a decoupled learning strategy for RL that creates a shared representation space where knowledge can be robustly transferred. We separate learning the task representation, the forward dynamics, the inverse dynamics and the reward function of the domain, and show that this decoupling improves performance within the task, transfers well to changes in dynamics and reward, and can be effectively used for online planning. Empirical results show good performance in both continuous and discrete RL domains.

References in corpus (8)

• Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)
• Benchmarking Deep Reinforcement Learning for Continuous Control
• Successor Features for Transfer in Reinforcement Learning
• Learning to Poke by Poking: Experiential Learning of Intuitive Physics
• Eigenoption Discovery through the Deep Successor Representation
• MazeBase: A Sandbox for Learning from Games
• Generalizing Skills with Semi-Supervised Reinforcement Learning
• Practical Learning of Predictive State Representations

Cited by in corpus (23)

• State Representation Learning for Control: An Overview
• Transfer Learning in Deep Reinforcement Learning: A Survey
• A survey on intrinsic motivation in reinforcement learning
• Single Episode Policy Transfer in Reinforcement Learning
• S-RL Toolbox: Environments, Datasets and Evaluation Metrics for State Representation Learning
• Strategies for Using Proximal Policy Optimization in Mobile Puzzle Games
• Can Increasing Input Dimensionality Improve Deep Reinforcement Learning?
• Hierarchically Organized Latent Modules for Exploratory Search in Morphogenetic Systems
• A General Framework for Structured Learning of Mechanical Systems
• An Empirical Study of Representation Learning for Reinforcement Learning in Healthcare
• Task-Agnostic Dynamics Priors for Deep Reinforcement Learning
• Disentangled Skill Embeddings for Reinforcement Learning
• Fast Adaptation via Policy-Dynamics Value Functions
• Which Mutual-Information Representation Learning Objectives are Sufficient for Control?
• Learning Markov State Abstractions for Deep Reinforcement Learning
• Agent Modelling under Partial Observability for Deep Reinforcement Learning
• Learning State Representations in Complex Systems with Multimodal Data
• REPAINT: Knowledge Transfer in Deep Reinforcement Learning
• Generalized Hidden Parameter MDPs Transferable Model-based RL in a Handful of Trials
• MANGA: Method Agnostic Neural-policy Generalization and Adaptation
• Fractional Transfer Learning for Deep Model-Based Reinforcement Learning
• Extracting Latent State Representations with Linear Dynamics from Rich Observations
• TAMPC: A Controller for Escaping Traps in Novel Environments