Hardware Conditioned Policies for Multi-Robot Transfer Learning

Abstract

Deep reinforcement learning could be used to learn dexterous robotic policies but it is challenging to transfer them to new robots with vastly different hardware properties. It is also prohibitively expensive to learn a new policy from scratch for each robot hardware due to the high sample complexity of modern state-of-the-art algorithms. We propose a novel approach called \textit{Hardware Conditioned Policies} where we train a universal policy conditioned on a vector representation of robot hardware. We considered robots in simulation with varied dynamics, kinematic structure, kinematic lengths and degrees-of-freedom. First, we use the kinematic structure directly as the hardware encoding and show great zero-shot transfer to completely novel robots not seen during training. For robots with lower zero-shot success rate, we also demonstrate that fine-tuning the policy network is significantly more sample-efficient than training a model from scratch. In tasks where knowing the agent dynamics is important for success, we learn an embedding for robot hardware and show that policies conditioned on the encoding of hardware tend to generalize and transfer well. The code and videos are available on the project webpage: https://sites.google.com/view/robot-transfer-hcp.

Cited by in corpus (13)

• Lifelong Incremental Reinforcement Learning with Online Bayesian Inference
• One Policy to Control Them All: Shared Modular Policies for Agent-Agnostic Control
• Instance Weighted Incremental Evolution Strategies for Reinforcement Learning in Dynamic Environments
• MULTIPOLAR: Multi-Source Policy Aggregation for Transfer Reinforcement Learning between Diverse Environmental Dynamics
• Dimensionless Policies based on the Buckingham $\pi$ Theorem: Is This a Good Way to Generalize Numerical Results?
• Skill Transfer in Deep Reinforcement Learning under Morphological Heterogeneity
• Learning Guidance Rewards with Trajectory-space Smoothing
• Harnessing Distribution Ratio Estimators for Learning Agents with Quality and Diversity
• My Body is a Cage: the Role of Morphology in Graph-Based Incompatible Control
• Hierarchically Decoupled Imitation for Morphological Transfer
• Learning Domain Randomization Distributions for Training Robust Locomotion Policies
• Emergent Hand Morphology and Control from Optimizing Robust Grasps of Diverse Objects
• Tool as Embodiment for Recursive Manipulation