Learning Plannable Representations with Causal InfoGAN

Abstract

In recent years, deep generative models have been shown to 'imagine' convincing high-dimensional observations such as images, audio, and even video, learning directly from raw data. In this work, we ask how to imagine goal-directed visual plans -- a plausible sequence of observations that transition a dynamical system from its current configuration to a desired goal state, which can later be used as a reference trajectory for control. We focus on systems with high-dimensional observations, such as images, and propose an approach that naturally combines representation learning and planning. Our framework learns a generative model of sequential observations, where the generative process is induced by a transition in a low-dimensional planning model, and an additional noise. By maximizing the mutual information between the generated observations and the transition in the planning model, we obtain a low-dimensional representation that best explains the causal nature of the data. We structure the planning model to be compatible with efficient planning algorithms, and we propose several such models based on either discrete or continuous states. Finally, to generate a visual plan, we project the current and goal observations onto their respective states in the planning model, plan a trajectory, and then use the generative model to transform the trajectory to a sequence of observations. We demonstrate our method on imagining plausible visual plans of rope manipulation.

arXiv commentICML / IJCAI / AAMAS 2018 Workshop on Planning and Learning (PAL-18)

References in corpus (15)

• Categorical Reparameterization with Gumbel-Softmax
• End-to-End Training of Deep Visuomotor Policies
• Improved Techniques for Training GANs
• InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets
• Adversarial Feature Learning
• A Recurrent Latent Variable Model for Sequential Data
• Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization
• Action-Conditional Video Prediction using Deep Networks in Atari Games
• Composing graphical models with neural networks for structured representations and fast inference
• Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Images
• Temporal Difference Models: Model-Free Deep RL for Model-Based Control
• Schema Networks: Zero-shot Transfer with a Generative Causal Model of Intuitive Physics
• Semi-parametric Topological Memory for Navigation
• Efficient Model-Based Deep Reinforcement Learning with Variational State Tabulation
• Value Prediction Network

Cited by in corpus (39)

• A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithms
• Learning Predictive Representations for Deformable Objects Using Contrastive Estimation
• Contrastive Learning of Structured World Models
• COBRA: Data-Efficient Model-Based RL through Unsupervised Object Discovery and Curiosity-Driven Exploration
• Causal Induction from Visual Observations for Goal Directed Tasks
• Long-Horizon Visual Planning with Goal-Conditioned Hierarchical Predictors
• Inverse reinforcement learning for video games
• Learning Neural-Symbolic Descriptive Planning Models via Cube-Space Priors: The Voyage Home (to STRIPS)
• Hallucinative Topological Memory for Zero-Shot Visual Planning
• Procedure Planning in Instructional Videos
• Prediction, Consistency, Curvature: Representation Learning for Locally-Linear Control
• Learning Semantic Embedding Spaces for Slicing Vegetables
• Example-Driven Model-Based Reinforcement Learning for Solving Long-Horizon Visuomotor Tasks
• Bayesian Relational Memory for Semantic Visual Navigation
• The Differentiable Cross-Entropy Method
• Learning Markov State Abstractions for Deep Reinforcement Learning
• Plan2Vec: Unsupervised Representation Learning by Latent Plans
• DeepAveragers: Offline Reinforcement Learning by Solving Derived Non-Parametric MDPs
• Towards Stable Symbol Grounding with Zero-Suppressed State AutoEncoder
• Predictive Coding for Locally-Linear Control
• Estimating Q(s,s') with Deep Deterministic Dynamics Gradients
• Agent Modelling under Partial Observability for Deep Reinforcement Learning
• Neural-Symbolic Descriptive Action Model from Images: The Search for STRIPS
• Hierarchies of Planning and Reinforcement Learning for Robot Navigation
• CLOUD: Contrastive Learning of Unsupervised Dynamics
• Learning Disentangled Representations of Video with Missing Data
• Modular Action Concept Grounding in Semantic Video Prediction
• Solving Physics Puzzles by Reasoning about Paths
• Building Bridges: Generative Artworks to Explore AI Ethics
• Learning Topological Motion Primitives for Knot Planning
• Discrete Word Embedding for Logical Natural Language Understanding
• Incremental Learning of Discrete Planning Domains from Continuous Perceptions
• Causal Future Prediction in a Minkowski Space-Time
• Control-Tutored Reinforcement Learning: an application to the Herding Problem
• Procedure Planning in Instructional Videos via Contextual Modeling and Model-based Policy Learning
• Learning What To Do by Simulating the Past
• Subgoal Search For Complex Reasoning Tasks
• Progressive growing of self-organized hierarchical representations for exploration
• Incremental learning abstract discrete planning domains and mappings to continuous perceptions