Learning Robust State Abstractions for Hidden-Parameter Block MDPs

Abstract

Many control tasks exhibit similar dynamics that can be modeled as having common latent structure. Hidden-Parameter Markov Decision Processes (HiP-MDPs) explicitly model this structure to improve sample efficiency in multi-task settings. However, this setting makes strong assumptions on the observability of the state that limit its application in real-world scenarios with rich observation spaces. In this work, we leverage ideas of common structure from the HiP-MDP setting, and extend it to enable robust state abstractions inspired by Block MDPs. We derive instantiations of this new framework for both multi-task reinforcement learning (MTRL) and meta-reinforcement learning (Meta-RL) settings. Further, we provide transfer and generalization bounds based on task and state similarity, along with sample complexity bounds that depend on the aggregate number of samples across tasks, rather than the number of tasks, a significant improvement over prior work that use the same environment assumptions. To further demonstrate the efficacy of the proposed method, we empirically compare and show improvement over multi-task and meta-reinforcement learning baselines.

arXiv commentAccepted at the 9th International Conference on Learning Representations. 22 pages, 14 figures

References in corpus (11)

• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• DeepMind Control Suite
• Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables
• Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees
• The Benefit of Multitask Representation Learning
• Meta-Learning by Adjusting Priors Based on Extended PAC-Bayes Theory
• Learning Invariant Representations for Reinforcement Learning without Reconstruction
• DeepMDP: Learning Continuous Latent Space Models for Representation Learning
• Sharing Knowledge in Multi-Task Deep Reinforcement Learning
• Sequential Transfer in Reinforcement Learning with a Generative Model

Cited by in corpus (3)

• Structure in Deep Reinforcement Learning: A Survey and Open Problems
• Learning Representations for Pixel-based Control: What Matters and Why?
• Learning Top-k Subtask Planning Tree based on Discriminative Representation Pre-training for Decision Making