Never Stop Learning: The Effectiveness of Fine-Tuning in Robotic Reinforcement Learning

Abstract

One of the great promises of robot learning systems is that they will be able to learn from their mistakes and continuously adapt to ever-changing environments. Despite this potential, most of the robot learning systems today are deployed as a fixed policy and they are not being adapted after their deployment. Can we efficiently adapt previously learned behaviors to new environments, objects and percepts in the real world? In this paper, we present a method and empirical evidence towards a robot learning framework that facilitates continuous adaption. In particular, we demonstrate how to adapt vision-based robotic manipulation policies to new variations by fine-tuning via off-policy reinforcement learning, including changes in background, object shape and appearance, lighting conditions, and robot morphology. Further, this adaptation uses less than 0.2% of the data necessary to learn the task from scratch. We find that our approach of adapting pre-trained policies leads to substantial performance gains over the course of fine-tuning, and that pre-training via RL is essential: training from scratch or adapting from supervised ImageNet features are both unsuccessful with such small amounts of data. We also find that these positive results hold in a limited continual learning setting, in which we repeatedly fine-tune a single lineage of policies using data from a succession of new tasks. Our empirical conclusions are consistently supported by experiments on simulated manipulation tasks, and by 52 unique fine-tuning experiments on a real robotic grasping system pre-trained on 580,000 grasps.

arXiv comment8.5 pages, 9 figures. See video overview and experiments at https://youtu.be/pPDVewcSpdc and project website at https://ryanjulian.me/continual-fine-tuning

References in corpus (22)

• DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition
• How transferable are features in deep neural networks?
• An Overview of Multi-Task Learning in Deep Neural Networks
• Robots that can adapt like animals
• Solving Rubik's Cube with a Robot Hand
• Recurrent World Models Facilitate Policy Evolution
• Visual Reinforcement Learning with Imagined Goals
• What makes ImageNet good for transfer learning?
• One-Shot Visual Imitation Learning via Meta-Learning
• Learning by Playing - Solving Sparse Reward Tasks from Scratch
• Reset-free Trial-and-Error Learning for Robot Damage Recovery
• Sim-to-Real Robot Learning from Pixels with Progressive Nets
• Deep Dynamics Models for Learning Dexterous Manipulation
• Robot Learning in Homes: Improving Generalization and Reducing Dataset Bias
• Deep Online Learning via Meta-Learning: Continual Adaptation for Model-Based RL
• Adaptive Prior Selection for Repertoire-based Online Adaptation in Robotics
• Task-Embedded Control Networks for Few-Shot Imitation Learning
• Modular meta-learning
• Off-Policy Evaluation via Off-Policy Classification
• Modelling Generalized Forces with Reinforcement Learning for Sim-to-Real Transfer
• Catch & Carry: Reusable Neural Controllers for Vision-Guided Whole-Body Tasks
• Experience-Embedded Visual Foresight

Cited by in corpus (10)

• Real-Time Deep Learning Approach to Visual Servo Control and Grasp Detection for Autonomous Robotic Manipulation
• Self-Supervised Policy Adaptation during Deployment
• COG: Connecting New Skills to Past Experience with Offline Reinforcement Learning
• Behavior From the Void: Unsupervised Active Pre-Training
• Heuristic-free Optimization of Force-Controlled Robot Search Strategies in Stochastic Environments
• Model-Based Domain Generalization
• A Simple Approach to Continual Learning by Transferring Skill Parameters
• A Geometric Perspective on Self-Supervised Policy Adaptation
• Transfer Reinforcement Learning across Homotopy Classes
• Improved Reinforcement Learning Coordinated Control of a Mobile Manipulator using Joint Clamping