Continual State Representation Learning for Reinforcement Learning using Generative Replay

Abstract

We consider the problem of building a state representation model in a continual fashion. As the environment changes, the aim is to efficiently compress the sensory state's information without losing past knowledge. The learned features are then fed to a Reinforcement Learning algorithm to learn a policy. We propose to use Variational Auto-Encoders for state representation, and Generative Replay, i.e. the use of generated samples, to maintain past knowledge. We also provide a general and statistically sound method for automatic environment change detection. Our method provides efficient state representation as well as forward transfer, and avoids catastrophic forgetting. The resulting model is capable of incrementally learning information without using past data and with a bounded system size.

arXiv commentAccepted contribution to the Workshop on Continual Learning, NeurIPS 2018 (Neural Information Processing Systems)

References in corpus (10)

• Auto-Encoding Variational Bayes
• Recurrent World Models Facilitate Policy Evolution
• State Representation Learning for Control: An Overview
• Variational Continual Learning
• Unsupervised Predictive Memory in a Goal-Directed Agent
• DARLA: Improving Zero-Shot Transfer in Reinforcement Learning
• Life-Long Disentangled Representation Learning with Cross-Domain Latent Homologies
• Memory Replay GANs: learning to generate images from new categories without forgetting
• S-RL Toolbox: Environments, Datasets and Evaluation Metrics for State Representation Learning
• Flatland: a Lightweight First-Person 2-D Environment for Reinforcement Learning

Cited by in corpus (8)

• Pseudo-Rehearsal: Achieving Deep Reinforcement Learning without Catastrophic Forgetting
• M2KD: Multi-model and Multi-level Knowledge Distillation for Incremental Learning
• Dynamics-aware Embeddings
• Predictive Coding for Boosting Deep Reinforcement Learning with Sparse Rewards
• GAN Memory with No Forgetting
• Fuzzy Tiling Activations: A Simple Approach to Learning Sparse Representations Online
• GRIm-RePR: Prioritising Generating Important Features for Pseudo-Rehearsal
• Progressive growing of self-organized hierarchical representations for exploration