Near-Optimal Representation Learning for Hierarchical Reinforcement Learning

Abstract

We study the problem of representation learning in goal-conditioned hierarchical reinforcement learning. In such hierarchical structures, a higher-level controller solves tasks by iteratively communicating goals which a lower-level policy is trained to reach. Accordingly, the choice of representation -- the mapping of observation space to goal space -- is crucial. To study this problem, we develop a notion of sub-optimality of a representation, defined in terms of expected reward of the optimal hierarchical policy using this representation. We derive expressions which bound the sub-optimality and show how these expressions can be translated to representation learning objectives which may be optimized in practice. Results on a number of difficult continuous-control tasks show that our approach to representation learning yields qualitatively better representations as well as quantitatively better hierarchical policies, compared to existing methods (see videos at https://sites.google.com/view/representation-hrl).

arXiv commentICLR 2019 Conference Paper

References in corpus (9)

• Continuous control with deep reinforcement learning
• Representation Learning with Contrastive Predictive Coding
• Trust Region Policy Optimization
• Learning deep representations by mutual information estimation and maximization
• MINE: Mutual Information Neural Estimation
• Early Visual Concept Learning with Unsupervised Deep Learning
• Constrained Policy Optimization
• Near Optimal Behavior via Approximate State Abstraction
• Model Reduction Techniques for Computing Approximately Optimal Solutions for Markov Decision Processes

Cited by in corpus (31)

• Why Does Hierarchy (Sometimes) Work So Well in Reinforcement Learning?
• An information-theoretic perspective on intrinsic motivation in reinforcement learning: a survey
• ReLMoGen: Leveraging Motion Generation in Reinforcement Learning for Mobile Manipulation
• Unsupervised State Representation Learning in Atari
• Representation Matters: Offline Pretraining for Sequential Decision Making
• Learning Compositional Neural Programs with Recursive Tree Search and Planning
• From Pixels to Legs: Hierarchical Learning of Quadruped Locomotion
• State Alignment-based Imitation Learning
• Dynamics-aware Embeddings
• OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learning
• Hierarchical Policy Learning is Sensitive to Goal Space Design
• C-Learning: Learning to Achieve Goals via Recursive Classification
• Data-efficient Hindsight Off-policy Option Learning
• Which Mutual-Information Representation Learning Objectives are Sufficient for Control?
• Provable Representation Learning for Imitation with Contrastive Fourier Features
• Mutual Information Maximization for Robust Plannable Representations
• An A* Curriculum Approach to Reinforcement Learning for RGBD Indoor Robot Navigation
• Stochastic Adversarial Koopman Model for Dynamical Systems
• Weakly-Supervised Reinforcement Learning for Controllable Behavior
• TRAIL: Near-Optimal Imitation Learning with Suboptimal Data
• Predictive Coding for Locally-Linear Control
• Predictive Coding for Boosting Deep Reinforcement Learning with Sparse Rewards
• Hierarchical reinforcement learning for efficient exploration and transfer
• Variational Empowerment as Representation Learning for Goal-Based Reinforcement Learning
• Hierarchies of Planning and Reinforcement Learning for Robot Navigation
• Hierarchically Decoupled Imitation for Morphological Transfer
• Inter-Level Cooperation in Hierarchical Reinforcement Learning
• Reinforcement Learning for Strategic Recommendations
• Hierarchical Representation Learning for Markov Decision Processes
• Efficient Robotic Object Search via HIEM: Hierarchical Policy Learning with Intrinsic-Extrinsic Modeling
• Hierarchical Reinforcement Learning with Optimal Level Synchronization Based on Flow-Based Deep Generative Model