MULTIPOLAR: Multi-Source Policy Aggregation for Transfer Reinforcement Learning between Diverse Environmental Dynamics

Abstract

Transfer reinforcement learning (RL) aims at improving the learning efficiency of an agent by exploiting knowledge from other source agents trained on relevant tasks. However, it remains challenging to transfer knowledge between different environmental dynamics without having access to the source environments. In this work, we explore a new challenge in transfer RL, where only a set of source policies collected under diverse unknown dynamics is available for learning a target task efficiently. To address this problem, the proposed approach, MULTI-source POLicy AggRegation (MULTIPOLAR), comprises two key techniques. We learn to aggregate the actions provided by the source policies adaptively to maximize the target task performance. Meanwhile, we learn an auxiliary network that predicts residuals around the aggregated actions, which ensures the target policy's expressiveness even when some of the source policies perform poorly. We demonstrated the effectiveness of MULTIPOLAR through an extensive experimental evaluation across six simulated environments ranging from classic control problems to challenging robotics simulations, under both continuous and discrete action spaces. The demo videos and code are available on the project webpage: https://omron-sinicx.github.io/multipolar/.

arXiv commentThis work was presented at IJCAI 2020. Copyright (c) 2020 International Joint Conferences on Artificial Intelligence, All rights reserved

References in corpus (10)

• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• An Introduction to Deep Reinforcement Learning
• Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation
• Sim-to-Real: Learning Agile Locomotion For Quadruped Robots
• Learning to Adapt in Dynamic, Real-World Environments Through Meta-Reinforcement Learning
• Policy Transfer with Strategy Optimization
• Hardware Conditioned Policies for Multi-Robot Transfer Learning
• Importance Weighted Transfer of Samples in Reinforcement Learning
• Hierarchical Reinforcement Learning via Advantage-Weighted Information Maximization
• Attend, Adapt and Transfer: Attentive Deep Architecture for Adaptive Transfer from multiple sources in the same domain

Cited by in corpus (2)

• Procedural Content Generation: Better Benchmarks for Transfer Reinforcement Learning
• Shared learning of powertrain control policies for vehicle fleets