Lifelong Incremental Reinforcement Learning with Online Bayesian Inference

Abstract

A central capability of a long-lived reinforcement learning (RL) agent is to incrementally adapt its behavior as its environment changes, and to incrementally build upon previous experiences to facilitate future learning in real-world scenarios. In this paper, we propose LifeLong Incremental Reinforcement Learning (LLIRL), a new incremental algorithm for efficient lifelong adaptation to dynamic environments. We develop and maintain a library that contains an infinite mixture of parameterized environment models, which is equivalent to clustering environment parameters in a latent space. The prior distribution over the mixture is formulated as a Chinese restaurant process (CRP), which incrementally instantiates new environment models without any external information to signal environmental changes in advance. During lifelong learning, we employ the expectation maximization (EM) algorithm with online Bayesian inference to update the mixture in a fully incremental manner. In EM, the E-step involves estimating the posterior expectation of environment-to-cluster assignments, while the M-step updates the environment parameters for future learning. This method allows for all environment models to be adapted as necessary, with new models instantiated for environmental changes and old models retrieved when previously seen environments are encountered again. Experiments demonstrate that LLIRL outperforms relevant existing methods, and enables effective incremental adaptation to various dynamic environments for lifelong learning.

arXiv commentAccepted by IEEE Transactions on Neural Networks and Learning Systems, 2021

References in corpus (15)

• Overcoming catastrophic forgetting in neural networks
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Learning agile and dynamic motor skills for legged robots
• Benchmarking Deep Reinforcement Learning for Continuous Control
• Sim-to-Real Transfer of Robotic Control with Dynamics Randomization
• Deep Drone Racing: From Simulation to Reality with Domain Randomization
• Continual Learning of Context-dependent Processing in Neural Networks
• On-line Policy Improvement using Monte-Carlo Search
• Assessing Transferability from Simulation to Reality for Reinforcement Learning
• Dynamic Evaluation of Neural Sequence Models
• Policy Transfer with Strategy Optimization
• Deep Online Learning via Meta-Learning: Continual Adaptation for Model-Based RL
• Hardware Conditioned Policies for Multi-Robot Transfer Learning
• Importance Weighted Transfer of Samples in Reinforcement Learning
• Safe Policy Search for Lifelong Reinforcement Learning with Sublinear Regret

Cited by in corpus (8)

• Automatic Speech Recognition using Advanced Deep Learning Approaches: A survey
• Rule-Based Reinforcement Learning for Efficient Robot Navigation with Space Reduction
• A Dirichlet Process Mixture of Robust Task Models for Scalable Lifelong Reinforcement Learning
• Instance Weighted Incremental Evolution Strategies for Reinforcement Learning in Dynamic Environments
• Depthwise Convolution for Multi-Agent Communication with Enhanced Mean-Field Approximation
• Magnetic Field-Based Reward Shaping for Goal-Conditioned Reinforcement Learning
• Efficient Bayesian Policy Reuse with a Scalable Observation Model in Deep Reinforcement Learning
• Growable and Interpretable Neural Control with Online Continual Learning for Autonomous Lifelong Locomotion Learning Machines