A Study of Continual Learning Methods for Q-Learning

Abstract

We present an empirical study on the use of continual learning (CL) methods in a reinforcement learning (RL) scenario, which, to the best of our knowledge, has not been described before. CL is a very active recent research topic concerned with machine learning under non-stationary data distributions. Although this naturally applies to RL, the use of dedicated CL methods is still uncommon. This may be due to the fact that CL methods often assume a decomposition of CL problems into disjoint sub-tasks of stationary distribution, that the onset of these sub-tasks is known, and that sub-tasks are non-contradictory. In this study, we perform an empirical comparison of selected CL methods in a RL problem where a physically simulated robot must follow a racetrack by vision. In order to make CL methods applicable, we restrict the RL setting and introduce non-conflicting subtasks of known onset, which are however not disjoint and whose distribution, from the learner's point of view, is still non-stationary. Our results show that dedicated CL methods can significantly improve learning when compared to the baseline technique of "experience replay".

Accepted at the IJCNN2022, 9 pages, 9 figures

References in corpus (7)

• PathNet: Evolution Channels Gradient Descent in Super Neural Networks
• Efficient Lifelong Learning with A-GEM
• On Tiny Episodic Memories in Continual Learning
• Optimal Continual Learning has Perfect Memory and is NP-hard
• An Investigation of Replay-based Approaches for Continual Learning
• Continual World: A Robotic Benchmark For Continual Reinforcement Learning
• CORA: Benchmarks, Baselines, and Metrics as a Platform for Continual Reinforcement Learning Agents

Cited by in corpus (1)

• Adapting the Behavior of Reinforcement Learning Agents to Changing Action Spaces and Reward Functions