A Simple Approach to Continual Learning by Transferring Skill Parameters

Abstract

In order to be effective general purpose machines in real world environments, robots not only will need to adapt their existing manipulation skills to new circumstances, they will need to acquire entirely new skills on-the-fly. A great promise of continual learning is to endow robots with this ability, by using their accumulated knowledge and experience from prior skills. We take a fresh look at this problem, by considering a setting in which the robot is limited to storing that knowledge and experience only in the form of learned skill policies. We show that storing skill policies, careful pre-training, and appropriately choosing when to transfer those skill policies is sufficient to build a continual learner in the context of robotic manipulation. We analyze which conditions are needed to transfer skills in the challenging Meta-World simulation benchmark. Using this analysis, we introduce a pair-wise metric relating skills that allows us to predict the effectiveness of skill transfer between tasks, and use it to reduce the problem of continual learning to curriculum selection. Given an appropriate curriculum, we show how to continually acquire robotic manipulation skills without forgetting, and using far fewer samples than needed to train them from scratch.

Submitted to ICRA 2022

References in corpus (37)

• Continuous control with deep reinforcement learning
• Playing Atari with Deep Reinforcement Learning
• DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition
• An Overview of Multi-Task Learning in Deep Neural Networks
• Robots that can adapt like animals
• Solving Rubik's Cube with a Robot Hand
• Recurrent World Models Facilitate Policy Evolution
• Learning to Navigate in Complex Environments
• One-Shot Visual Imitation Learning via Meta-Learning
• Reset-free Trial-and-Error Learning for Robot Damage Recovery
• Sim-to-Real Robot Learning from Pixels with Progressive Nets
• Learning to Adapt in Dynamic, Real-World Environments Through Meta-Reinforcement Learning
• AWAC: Accelerating Online Reinforcement Learning with Offline Datasets
• Iterative Reinforcement Learning Based Design of Dynamic Locomotion Skills for Cassie
• Robot Learning in Homes: Improving Generalization and Reducing Dataset Bias
• Multi-Task Reinforcement Learning with Soft Modularization
• Skew-Fit: State-Covering Self-Supervised Reinforcement Learning
• Residual Policy Learning
• Mid-Level Visual Representations Improve Generalization and Sample Efficiency for Learning Visuomotor Policies
• Never Stop Learning: The Effectiveness of Fine-Tuning in Robotic Reinforcement Learning
• Learning to Walk via Deep Reinforcement Learning
• COG: Connecting New Skills to Past Experience with Offline Reinforcement Learning
• DisCoRL: Continual Reinforcement Learning via Policy Distillation
• Adaptive Prior Selection for Repertoire-based Online Adaptation in Robotics
• Task-Embedded Control Networks for Few-Shot Imitation Learning
• Modular meta-learning
• Broadly-Exploring, Local-Policy Trees for Long-Horizon Task Planning
• Modelling Generalized Forces with Reinforcement Learning for Sim-to-Real Transfer
• Experience-Embedded Visual Foresight
• Behavior Priors for Efficient Reinforcement Learning
• Reset-Free Lifelong Learning with Skill-Space Planning
• Data-efficient Hindsight Off-policy Option Learning
• Solving Compositional Reinforcement Learning Problems via Task Reduction
• Disentangled Planning and Control in Vision Based Robotics via Reward Machines
• Learning Task Decomposition with Ordered Memory Policy Network
• Information-theoretic Task Selection for Meta-Reinforcement Learning
• "What, not how": Solving an under-actuated insertion task from scratch