Lifelong Robotic Reinforcement Learning by Retaining Experiences

Abstract

Multi-task learning ideally allows robots to acquire a diverse repertoire of useful skills. However, many multi-task reinforcement learning efforts assume the robot can collect data from all tasks at all times. In reality, the tasks that the robot learns arrive sequentially, depending on the user and the robot's current environment. In this work, we study a practical sequential multi-task RL problem that is motivated by the practical constraints of physical robotic systems, and derive an approach that effectively leverages the data and policies learned for previous tasks to cumulatively grow the robot's skill-set. In a series of simulated robotic manipulation experiments, our approach requires less than half the samples than learning each task from scratch, while avoiding impractical round-robin data collection. On a Franka Emika Panda robot arm, our approach incrementally learns ten challenging tasks, including bottle capping and block insertion.

arXiv commentSupplementary website at https://sites.google.com/view/retain-experience/

References in corpus (18)

• Overcoming catastrophic forgetting in neural networks
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation
• Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization
• Hindsight Experience Replay
• Maximum Entropy Deep Inverse Reinforcement Learning
• Progress & Compress: A scalable framework for continual learning
• Learning by Playing - Solving Sparse Reward Tasks from Scratch
• robosuite: A Modular Simulation Framework and Benchmark for Robot Learning
• Policy Distillation
• Variational Inverse Control with Events: A General Framework for Data-Driven Reward Definition
• Kickstarting Deep Reinforcement Learning
• Learning to Walk via Deep Reinforcement Learning
• MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scale
• Few-Shot Goal Inference for Visuomotor Learning and Planning
• End-to-End Robotic Reinforcement Learning without Reward Engineering
• Importance Weighted Transfer of Samples in Reinforcement Learning
• REPAINT: Knowledge Transfer in Deep Reinforcement Learning