A Dirichlet Process Mixture of Robust Task Models for Scalable Lifelong Reinforcement Learning

Abstract

While reinforcement learning (RL) algorithms are achieving state-of-the-art performance in various challenging tasks, they can easily encounter catastrophic forgetting or interference when faced with lifelong streaming information. In the paper, we propose a scalable lifelong RL method that dynamically expands the network capacity to accommodate new knowledge while preventing past memories from being perturbed. We use a Dirichlet process mixture to model the non-stationary task distribution, which captures task relatedness by estimating the likelihood of task-to-cluster assignments and clusters the task models in a latent space. We formulate the prior distribution of the mixture as a Chinese restaurant process (CRP) that instantiates new mixture components as needed. The update and expansion of the mixture are governed by the Bayesian non-parametric framework with an expectation maximization (EM) procedure, which dynamically adapts the model complexity without explicit task boundaries or heuristics. Moreover, we use the domain randomization technique to train robust prior parameters for the initialization of each task model in the mixture, thus the resulting model can better generalize and adapt to unseen tasks. With extensive experiments conducted on robot navigation and locomotion domains, we show that our method successfully facilitates scalable lifelong RL and outperforms relevant existing methods.

arXiv commentManuscript accepted by IEEE Transactions on Cybernetics, 2022, DOI: DOI: 10.1109/TCYB.2022.3170485

References in corpus (12)

• Auto-Encoding Variational Bayes
• Overcoming catastrophic forgetting in neural networks
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Progressive Neural Networks
• Experience Replay for Continual Learning
• On Tiny Episodic Memories in Continual Learning
• Progress & Compress: A scalable framework for continual learning
• A Neural Dirichlet Process Mixture Model for Task-Free Continual Learning
• Assessing Transferability from Simulation to Reality for Reinforcement Learning
• Lifelong Incremental Reinforcement Learning with Online Bayesian Inference
• Dynamic Evaluation of Neural Sequence Models
• Policy Transfer with Strategy Optimization

Cited by in corpus (2)

• Depthwise Convolution for Multi-Agent Communication with Enhanced Mean-Field Approximation
• Magnetic Field-Based Reward Shaping for Goal-Conditioned Reinforcement Learning