Deep Reinforcement Learning amidst Lifelong Non-Stationarity

Abstract

As humans, our goals and our environment are persistently changing throughout our lifetime based on our experiences, actions, and internal and external drives. In contrast, typical reinforcement learning problem set-ups consider decision processes that are stationary across episodes. Can we develop reinforcement learning algorithms that can cope with the persistent change in the former, more realistic problem settings? While on-policy algorithms such as policy gradients in principle can be extended to non-stationary settings, the same cannot be said for more efficient off-policy algorithms that replay past experiences when learning. In this work, we formalize this problem setting, and draw upon ideas from the online learning and probabilistic inference literature to derive an off-policy RL algorithm that can reason about and tackle such lifelong non-stationarity. Our method leverages latent variable models to learn a representation of the environment from current and past experiences, and performs off-policy RL with this representation. We further introduce several simulation environments that exhibit lifelong non-stationarity, and empirically find that our approach substantially outperforms approaches that do not reason about environment shift.

arXiv commentsupplementary website at https://sites.google.com/stanford.edu/lilac/

References in corpus (19)

• Auto-Encoding Variational Bayes
• Overcoming catastrophic forgetting in neural networks
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Continual Learning Through Synaptic Intelligence
• A Simple Neural Attentive Meta-Learner
• Bayesian Online Changepoint Detection
• RL${}^2$: Fast Reinforcement Learning via Slow Reinforcement Learning
• Reinforcement Learning with Deep Energy-Based Policies
• Learning to reinforcement learn
• Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review
• Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning
• Learning Latent Dynamics for Planning from Pixels
• Finding Approximate POMDP solutions Through Belief Compression
• Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables
• Fast Context Adaptation via Meta-Learning
• Some Considerations on Learning to Explore via Meta-Reinforcement Learning
• Taming the Noise in Reinforcement Learning via Soft Updates
• Continuous Meta-Learning without Tasks
• Optimizing for the Future in Non-Stationary MDPs

Cited by in corpus (8)

• Learning Latent Representations to Influence Multi-Agent Interaction
• Reset-Free Lifelong Learning with Skill-Space Planning
• Towards Safe Policy Improvement for Non-Stationary MDPs
• Optimizing for the Future in Non-Stationary MDPs
• Sequoia: A Software Framework to Unify Continual Learning Research
• Improving Generalization in Meta-RL with Imaginary Tasks from Latent Dynamics Mixture
• Universal Off-Policy Evaluation
• Online hyperparameter optimization by real-time recurrent learning