Predictive Coding for Locally-Linear Control

Abstract

High-dimensional observations and unknown dynamics are major challenges when applying optimal control to many real-world decision making tasks. The Learning Controllable Embedding (LCE) framework addresses these challenges by embedding the observations into a lower dimensional latent space, estimating the latent dynamics, and then performing control directly in the latent space. To ensure the learned latent dynamics are predictive of next-observations, all existing LCE approaches decode back into the observation space and explicitly perform next-observation prediction---a challenging high-dimensional task that furthermore introduces a large number of nuisance parameters (i.e., the decoder) which are discarded during control. In this paper, we propose a novel information-theoretic LCE approach and show theoretically that explicit next-observation prediction can be replaced with predictive coding. We then use predictive coding to develop a decoder-free LCE model whose latent dynamics are amenable to locally-linear control. Extensive experiments on benchmark tasks show that our model reliably learns a controllable latent space that leads to superior performance when compared with state-of-the-art LCE baselines.

References in corpus (11)

• Adam: A Method for Stochastic Optimization
• Playing Atari with Deep Reinforcement Learning
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• Estimating divergence functionals and the likelihood ratio by convex risk minimization
• Model-Based Reinforcement Learning for Atari
• SOLAR: Deep Structured Representations for Model-Based Reinforcement Learning
• Learning Plannable Representations with Causal InfoGAN
• Safe Policy Improvement by Minimizing Robust Baseline Regret
• Near-Optimal Representation Learning for Hierarchical Reinforcement Learning
• Noise Contrastive Estimation and Negative Sampling for Conditional Models: Consistency and Statistical Efficiency
• Predictive Coding for Boosting Deep Reinforcement Learning with Sparse Rewards

Cited by in corpus (5)

• Control-Aware Representations for Model-based Reinforcement Learning
• Which Mutual-Information Representation Learning Objectives are Sufficient for Control?
• Action-Sufficient State Representation Learning for Control with Structural Constraints
• Learning the Linear Quadratic Regulator from Nonlinear Observations
• Learn Proportional Derivative Controllable Latent Space from Pixels