Transfer Reinforcement Learning across Homotopy Classes

Abstract

The ability for robots to transfer their learned knowledge to new tasks -- where data is scarce -- is a fundamental challenge for successful robot learning. While fine-tuning has been well-studied as a simple but effective transfer approach in the context of supervised learning, it is not as well-explored in the context of reinforcement learning. In this work, we study the problem of fine-tuning in transfer reinforcement learning when tasks are parameterized by their reward functions, which are known beforehand. We conjecture that fine-tuning drastically underperforms when source and target trajectories are part of different homotopy classes. We demonstrate that fine-tuning policy parameters across homotopy classes compared to fine-tuning within a homotopy class requires more interaction with the environment, and in certain cases is impossible. We propose a novel fine-tuning algorithm, Ease-In-Ease-Out fine-tuning, that consists of a relaxing stage and a curriculum learning stage to enable transfer learning across homotopy classes. Finally, we evaluate our approach on several robotics-inspired simulated environments and empirically verify that the Ease-In-Ease-Out fine-tuning method can successfully fine-tune in a sample-efficient way compared to existing baselines.

arXiv commentAccepted by IEEE Robotics and Automation Letters 2021

References in corpus (15)

• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Playing Atari with Deep Reinforcement Learning
• How transferable are features in deep neural networks?
• An Overview of Multi-Task Learning in Deep Neural Networks
• QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation
• RL${}^2$: Fast Reinforcement Learning via Slow Reinforcement Learning
• Learning to reinforcement learn
• Quantifying Generalization in Reinforcement Learning
• Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learning
• Safe Exploration in Finite Markov Decision Processes with Gaussian Processes
• Explicit Inductive Bias for Transfer Learning with Convolutional Networks
• Learning to Learn: Meta-Critic Networks for Sample Efficient Learning
• iGibson 1.0: a Simulation Environment for Interactive Tasks in Large Realistic Scenes
• Never Stop Learning: The Effectiveness of Fine-Tuning in Robotic Reinforcement Learning
• RoboNet: Large-Scale Multi-Robot Learning