Learning Invariant Feature Spaces to Transfer Skills with Reinforcement Learning

Abstract

People can learn a wide range of tasks from their own experience, but can also learn from observing other creatures. This can accelerate acquisition of new skills even when the observed agent differs substantially from the learning agent in terms of morphology. In this paper, we examine how reinforcement learning algorithms can transfer knowledge between morphologically different agents (e.g., different robots). We introduce a problem formulation where two agents are tasked with learning multiple skills by sharing information. Our method uses the skills that were learned by both agents to train invariant feature spaces that can then be used to transfer other skills from one agent to another. The process of learning these invariant feature spaces can be viewed as a kind of "analogy making", or implicit learning of partial correspondences between two distinct domains. We evaluate our transfer learning algorithm in two simulated robotic manipulation skills, and illustrate that we can transfer knowledge between simulated robotic arms with different numbers of links, as well as simulated arms with different actuation mechanisms, where one robot is torque-driven while the other is tendon-driven.

arXiv commentPublished as a conference paper at ICLR 2017

Cited by in corpus (42)

• Solving Rubik's Cube with a Robot Hand
• Deep Reinforcement Learning: An Overview
• An Algorithmic Perspective on Imitation Learning
• Deep Drone Racing: From Simulation to Reality with Domain Randomization
• One-Shot Visual Imitation Learning via Meta-Learning
• Learning Dexterous In-Hand Manipulation
• One-Shot Imitation Learning
• Transfer Learning in Deep Reinforcement Learning: A Survey
• Transferring End-to-End Visuomotor Control from Simulation to Real World for a Multi-Stage Task
• DARLA: Improving Zero-Shot Transfer in Reinforcement Learning
• A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithms
• Generative Adversarial Imitation from Observation
• One for Many: Transfer Learning for Building HVAC Control
• Transfer Learning for Future Wireless Networks: A Comprehensive Survey
• Imitation from Observation: Learning to Imitate Behaviors from Raw Video via Context Translation
• ManiSkill: Generalizable Manipulation Skill Benchmark with Large-Scale Demonstrations
• Exploiting Hierarchy for Learning and Transfer in KL-regularized RL
• Structure in Deep Reinforcement Learning: A Survey and Open Problems
• Benchmark Environments for Multitask Learning in Continuous Domains
• RoboNet: Large-Scale Multi-Robot Learning
• Recent Advances in Leveraging Human Guidance for Sequential Decision-Making Tasks
• ACNMP: Skill Transfer and Task Extrapolation through Learning from Demonstration and Reinforcement Learning via Representation Sharing
• State-only Imitation with Transition Dynamics Mismatch
• XIRL: Cross-embodiment Inverse Reinforcement Learning
• Context-Aware Policy Reuse
• Robust Visual Domain Randomization for Reinforcement Learning
• The Natural Language of Actions
• Robust Domain Randomised Reinforcement Learning through Peer-to-Peer Distillation
• Leveraging Human Guidance for Deep Reinforcement Learning Tasks
• Skill Transfer in Deep Reinforcement Learning under Morphological Heterogeneity
• Domain-Robust Visual Imitation Learning with Mutual Information Constraints
• Linear Representation Meta-Reinforcement Learning for Instant Adaptation
• Evaluating the progress of Deep Reinforcement Learning in the real world: aligning domain-agnostic and domain-specific research
• Bridging the Imitation Gap by Adaptive Insubordination
• Zero-Shot Learning of Text Adventure Games with Sentence-Level Semantics
• Cross-Domain Imitation Learning with a Dual Structure
• Cross-domain Imitation from Observations
• Reparameterized Variational Divergence Minimization for Stable Imitation
• Knowledge Transfer Between Robots with Similar Dynamics for High-Accuracy Impromptu Trajectory Tracking
• Targeted Deep Learning: Framework, Methods, and Applications
• Meta Adaptation using Importance Weighted Demonstrations
• Policy Transfer across Visual and Dynamics Domain Gaps via Iterative Grounding