Continual Reinforcement Learning with Diversity Exploration and Adversarial Self-Correction

Abstract

Deep reinforcement learning has made significant progress in the field of continuous control, such as physical control and autonomous driving. However, it is challenging for a reinforcement model to learn a policy for each task sequentially due to catastrophic forgetting. Specifically, the model would forget knowledge it learned in the past when trained on a new task. We consider this challenge from two perspectives: i) acquiring task-specific skills is difficult since task information and rewards are not highly related; ii) learning knowledge from previous experience is difficult in continuous control domains. In this paper, we introduce an end-to-end framework namely Continual Diversity Adversarial Network (CDAN). We first develop an unsupervised diversity exploration method to learn task-specific skills using an unsupervised objective. Then, we propose an adversarial self-correction mechanism to learn knowledge by exploiting past experience. The two learning procedures are presumably reciprocal. To evaluate the proposed method, we propose a new continuous reinforcement learning environment named Continual Ant Maze (CAM) and a new metric termed Normalized Shorten Distance (NSD). The experimental results confirm the effectiveness of diversity exploration and self-correction. It is worthwhile noting that our final result outperforms baseline by 18.35% in terms of NSD, and 0.61 according to the average reward.

References in corpus (18)

• Continuous control with deep reinforcement learning
• Playing Atari with Deep Reinforcement Learning
• Trust Region Policy Optimization
• Prioritized Experience Replay
• Dueling Network Architectures for Deep Reinforcement Learning
• An Introduction to Deep Reinforcement Learning
• PathNet: Evolution Channels Gradient Descent in Super Neural Networks
• DeepMind Control Suite
• Building Generalizable Agents with a Realistic and Rich 3D Environment
• Variational Intrinsic Control
• Emergent Complexity via Multi-Agent Competition
• Learning to Learn without Forgetting by Maximizing Transfer and Minimizing Interference
• HoME: a Household Multimodal Environment
• Variational Option Discovery Algorithms
• TStarBots: Defeating the Cheating Level Builtin AI in StarCraft II in the Full Game
• Deep Reinforcement Learning for Autonomous Driving
• Continual Reinforcement Learning with Complex Synapses
• Marathon Environments: Multi-Agent Continuous Control Benchmarks in a Modern Video Game Engine