Self-Consistent Trajectory Autoencoder: Hierarchical Reinforcement Learning with Trajectory Embeddings

Abstract

In this work, we take a representation learning perspective on hierarchical reinforcement learning, where the problem of learning lower layers in a hierarchy is transformed into the problem of learning trajectory-level generative models. We show that we can learn continuous latent representations of trajectories, which are effective in solving temporally extended and multi-stage problems. Our proposed model, SeCTAR, draws inspiration from variational autoencoders, and learns latent representations of trajectories. A key component of this method is to learn both a latent-conditioned policy and a latent-conditioned model which are consistent with each other. Given the same latent, the policy generates a trajectory which should match the trajectory predicted by the model. This model provides a built-in prediction mechanism, by predicting the outcome of closed loop policy behavior. We propose a novel algorithm for performing hierarchical RL with this model, combining model-based planning in the learned latent space with an unsupervised exploration objective. We show that our model is effective at reasoning over long horizons with sparse rewards for several simulated tasks, outperforming standard reinforcement learning methods and prior methods for hierarchical reasoning, model-based planning, and exploration.

arXiv commentAccepted at ICML 2018

References in corpus (10)

• VIME: Variational Information Maximizing Exploration
• Incentivizing Exploration In Reinforcement Learning With Deep Predictive Models
• Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning
• Unifying Count-Based Exploration and Intrinsic Motivation
• Stochastic Neural Networks for Hierarchical Reinforcement Learning
• Equivalence Between Policy Gradients and Soft Q-Learning
• Variational Intrinsic Control
• Meta Learning Shared Hierarchies
• Learning and Transfer of Modulated Locomotor Controllers
• Prediction and Control with Temporal Segment Models

Cited by in corpus (22)

• A survey on intrinsic motivation in reinforcement learning
• Efficient Exploration via State Marginal Matching
• Federated Deep Reinforcement Learning
• Unsupervised Meta-Learning for Reinforcement Learning
• An information-theoretic perspective on intrinsic motivation in reinforcement learning: a survey
• On Reward-Free Reinforcement Learning with Linear Function Approximation
• Dynamics Learning with Cascaded Variational Inference for Multi-Step Manipulation
• Learning Dynamics Model in Reinforcement Learning by Incorporating the Long Term Future
• Dynamics-aware Embeddings
• Shaping Belief States with Generative Environment Models for RL
• Fast Adaptation via Policy-Dynamics Value Functions
• The Differentiable Cross-Entropy Method
• Robot Motion Planning in Learned Latent Spaces
• Tutorial and Survey on Probabilistic Graphical Model and Variational Inference in Deep Reinforcement Learning
• AVGCN: Trajectory Prediction using Graph Convolutional Networks Guided by Human Attention
• Abstract Value Iteration for Hierarchical Reinforcement Learning
• Autonomous Vehicles Drive into Shared Spaces: eHMI Design Concept Focusing on Vulnerable Road Users
• Learning Calibratable Policies using Programmatic Style-Consistency
• Trajectory saliency detection using consistency-oriented latent codes from a recurrent auto-encoder
• Hierarchical Reinforcement Learning with Optimal Level Synchronization Based on Flow-Based Deep Generative Model
• PANDA: A Gigapixel-level Human-centric Video Dataset
• From abstract items to latent spaces to observed data and back: Compositional Variational Auto-Encoder