Block Contextual MDPs for Continual Learning
arXiv:2110.06972
Abstract
In reinforcement learning (RL), when defining a Markov Decision Process (MDP), the environment dynamics is implicitly assumed to be stationary. This assumption of stationarity, while simplifying, can be unrealistic in many scenarios. In the continual reinforcement learning scenario, the sequence of tasks is another source of nonstationarity. In this work, we propose to examine this continual reinforcement learning setting through the block contextual MDP (BC-MDP) framework, which enables us to relax the assumption of stationarity. This framework challenges RL algorithms to handle both nonstationarity and rich observation settings and, by additionally leveraging smoothness properties, enables us to study generalization bounds for this setting. Finally, we take inspiration from adaptive control to propose a novel algorithm that addresses the challenges introduced by this more realistic BC-MDP setting, allows for zero-shot adaptation at evaluation time, and achieves strong performance on several nonstationary environments.
arXiv comment26pages, Under Review
References in corpus (28)
- Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
- DeepMind Control Suite
- Gradient Episodic Memory for Continual Learning
- Learning Latent Dynamics for Planning from Pixels
- On Tiny Episodic Memories in Continual Learning
- Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control
- Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables
- Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Images
- Modular Multitask Reinforcement Learning with Policy Sketches
- Reinforced Continual Learning
- Successor Features for Transfer in Reinforcement Learning
- Fast Context Adaptation via Meta-Learning
- Transfer Learning in Deep Reinforcement Learning: A Survey
- EPOpt: Learning Robust Neural Network Policies Using Model Ensembles
- Learning Visual Predictive Models of Physics for Playing Billiards
- Online Continual Learning with Maximally Interfered Retrieval
- Hypernetwork functional image representation
- Meta reinforcement learning as task inference
- Provably efficient RL with Rich Observations via Latent State Decoding
- Hidden Parameter Markov Decision Processes: A Semiparametric Regression Approach for Discovering Latent Task Parametrizations
- Environment Probing Interaction Policies
- Principled Weight Initialization for Hypernetworks
- Meta-Learning Representations for Continual Learning
- Variational Recurrent Models for Solving Partially Observable Control Tasks
- Multi-Task Reinforcement Learning with Context-based Representations
- DensePhysNet: Learning Dense Physical Object Representations via Multi-step Dynamic Interactions
- Learning Adaptive Exploration Strategies in Dynamic Environments Through Informed Policy Regularization
- Modular Universal Reparameterization: Deep Multi-task Learning Across Diverse Domains