Lifetime policy reuse and the importance of task capacity

Abstract

A long-standing challenge in artificial intelligence is lifelong reinforcement learning, where learners are given many tasks in sequence and must transfer knowledge between tasks while avoiding catastrophic forgetting. Policy reuse and other multi-policy reinforcement learning techniques can learn multiple tasks but may generate many policies. This paper presents two novel contributions, namely 1) Lifetime Policy Reuse, a model-agnostic policy reuse algorithm that avoids generating many policies by optimising a fixed number of near-optimal policies through a combination of policy optimisation and adaptive policy selection; and 2) the task capacity, a measure for the maximal number of tasks that a policy can accurately solve. Comparing two state-of-the-art base-learners, the results demonstrate the importance of Lifetime Policy Reuse and task capacity based pre-selection on an 18-task partially observable Pacman domain and a Cartpole domain of up to 125 tasks.

References in corpus (17)

• Overcoming catastrophic forgetting in neural networks
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• ADADELTA: An Adaptive Learning Rate Method
• High-Dimensional Continuous Control Using Generalized Advantage Estimation
• Robots that can adapt like animals
• Emergence of Locomotion Behaviours in Rich Environments
• Large-Scale Study of Curiosity-Driven Learning
• Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning
• A Deep Hierarchical Approach to Lifelong Learning in Minecraft
• Playing FPS Games with Deep Reinforcement Learning
• Learning to Learn without Forgetting by Maximizing Transfer and Minimizing Interference
• Gotta Learn Fast: A New Benchmark for Generalization in RL
• Learning Multi-Level Hierarchies with Hindsight
• Hierarchical Reinforcement Learning with Advantage-Based Auxiliary Rewards
• Discounted Reinforcement Learning Is Not an Optimization Problem
• Reinforcement Learning for Non-Stationary Markov Decision Processes: The Blessing of (More) Optimism
• An Optimal Online Method of Selecting Source Policies for Reinforcement Learning