Behavior Self-Organization Supports Task Inference for Continual Robot Learning

Abstract

Recent advances in robot learning have enabled robots to become increasingly better at mastering a predefined set of tasks. On the other hand, as humans, we have the ability to learn a growing set of tasks over our lifetime. Continual robot learning is an emerging research direction with the goal of endowing robots with this ability. In order to learn new tasks over time, the robot first needs to infer the task at hand. Task inference, however, has received little attention in the multi-task learning literature. In this paper, we propose a novel approach to continual learning of robotic control tasks. Our approach performs unsupervised learning of behavior embeddings by incrementally self-organizing demonstrated behaviors. Task inference is made by finding the nearest behavior embedding to a demonstrated behavior, which is used together with the environment state as input to a multi-task policy trained with reinforcement learning to optimize performance over tasks. Unlike previous approaches, our approach makes no assumptions about task distribution and requires no task exploration to infer tasks. We evaluate our approach in experiments with concurrently and sequentially presented tasks and show that it outperforms other multi-task learning approaches in terms of generalization performance and convergence speed, particularly in the continual learning setting.

arXiv commentAccepted at IROS 2021

References in corpus (17)

• Adam: A Method for Stochastic Optimization
• Auto-Encoding Variational Bayes
• Continuous control with deep reinforcement learning
• TensorFlow: A system for large-scale machine learning
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• End-to-End Training of Deep Visuomotor Policies
• Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model
• Sim-to-Real Transfer of Robotic Control with Dynamics Randomization
• A Simple Neural Attentive Meta-Learner
• Active Learning of Inverse Models with Intrinsically Motivated Goal Exploration in Robots
• Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables
• Evolved Policy Gradients
• Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning
• Policy Distillation
• On-Policy Robot Imitation Learning from a Converging Supervisor
• Single Episode Policy Transfer in Reinforcement Learning
• A Meta-MDP Approach to Exploration for Lifelong Reinforcement Learning

Cited by in corpus (3)

• Continual Robot Learning using Self-Supervised Task Inference
• Map-based Experience Replay: A Memory-Efficient Solution to Catastrophic Forgetting in Reinforcement Learning
• Contrast, Imitate, Adapt: Learning Robotic Skills From Raw Human Videos