Continual World: A Robotic Benchmark For Continual Reinforcement Learning

Abstract

Continual learning (CL) -- the ability to continuously learn, building on previously acquired knowledge -- is a natural requirement for long-lived autonomous reinforcement learning (RL) agents. While building such agents, one needs to balance opposing desiderata, such as constraints on capacity and compute, the ability to not catastrophically forget, and to exhibit positive transfer on new tasks. Understanding the right trade-off is conceptually and computationally challenging, which we argue has led the community to overly focus on catastrophic forgetting. In response to these issues, we advocate for the need to prioritize forward transfer and propose Continual World, a benchmark consisting of realistic and meaningfully diverse robotic tasks built on top of Meta-World as a testbed. Following an in-depth empirical evaluation of existing CL methods, we pinpoint their limitations and highlight unique algorithmic challenges in the RL setting. Our benchmark aims to provide a meaningful and computationally inexpensive challenge for the community and thus help better understand the performance of existing and future solutions. Information about the benchmark, including the open-source code, is available at https://sites.google.com/view/continualworld.

arXiv commentNeurIPS 2021

References in corpus (20)

• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Soft Actor-Critic Algorithms and Applications
• A continual learning survey: Defying forgetting in classification tasks
• Zero-Shot Text-to-Image Generation
• Efficient Lifelong Learning with A-GEM
• Layer Normalization
• Towards Robust Evaluations of Continual Learning
• LAMOL: LAnguage MOdeling for Lifelong Language Learning
• On Quadratic Penalties in Elastic Weight Consolidation
• Don't forget, there is more than forgetting: new metrics for Continual Learning
• Adapting Auxiliary Losses Using Gradient Similarity
• Continual Unsupervised Representation Learning
• Ray Interference: a Source of Plateaus in Deep Reinforcement Learning
• CausalWorld: A Robotic Manipulation Benchmark for Causal Structure and Transfer Learning
• Lifelong Policy Gradient Learning of Factored Policies for Faster Training Without Forgetting
• Policy Consolidation for Continual Reinforcement Learning
• Adaptive Online Planning for Continual Lifelong Learning
• Evaluating Online Continual Learning with CALM
• Jelly Bean World: A Testbed for Never-Ending Learning

Cited by in corpus (4)

• CORA: Benchmarks, Baselines, and Metrics as a Platform for Continual Reinforcement Learning Agents
• Powerpropagation: A sparsity inducing weight reparameterisation
• Learning Vision-based Robotic Manipulation Tasks Sequentially in Offline Reinforcement Learning Settings
• A Study of Continual Learning Methods for Q-Learning