Neural Predictive Belief Representations

Abstract

Unsupervised representation learning has succeeded with excellent results in many applications. It is an especially powerful tool to learn a good representation of environments with partial or noisy observations. In partially observable domains it is important for the representation to encode a belief state, a sufficient statistic of the observations seen so far. In this paper, we investigate whether it is possible to learn such a belief representation using modern neural architectures. Specifically, we focus on one-step frame prediction and two variants of contrastive predictive coding (CPC) as the objective functions to learn the representations. To evaluate these learned representations, we test how well they can predict various pieces of information about the underlying state of the environment, e.g., position of the agent in a 3D maze. We show that all three methods are able to learn belief representations of the environment, they encode not only the state information, but also its uncertainty, a crucial aspect of belief states. We also find that for CPC multi-step predictions and action-conditioning are critical for accurate belief representations in visually complex environments. The ability of neural representations to capture the belief information has the potential to spur new advances for learning and planning in partially observable domains, where leveraging uncertainty is essential for optimal decision making.

References in corpus (11)

• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting
• Representation Learning with Contrastive Predictive Coding
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• Decision-Theoretic Planning: Structural Assumptions and Computational Leverage
• Unsupervised Predictive Memory in a Goal-Directed Agent
• Recurrent Reinforcement Learning: A Hybrid Approach
• Learning is planning: near Bayes-optimal reinforcement learning via Monte-Carlo tree search
• Learning Awareness Models
• Recurrent Predictive State Policy Networks
• Integrating Algorithmic Planning and Deep Learning for Partially Observable Navigation

Cited by in corpus (25)

• Contrastive Representation Learning: A Framework and Review
• Dream to Control: Learning Behaviors by Latent Imagination
• Decoupling Representation Learning from Reinforcement Learning
• Meta reinforcement learning as task inference
• Bootstrap Latent-Predictive Representations for Multitask Reinforcement Learning
• Pretraining Representations for Data-Efficient Reinforcement Learning
• A Survey of Embodied AI: From Simulators to Research Tasks
• Action and Perception as Divergence Minimization
• Discriminative Particle Filter Reinforcement Learning for Complex Partial Observations
• Causally Correct Partial Models for Reinforcement Learning
• Auxiliary Tasks and Exploration Enable ObjectNav
• Learning Belief Representations for Imitation Learning in POMDPs
• World Discovery Models
• Offline Meta Learning of Exploration
• Mutual Information Maximization for Robust Plannable Representations
• Neural Recursive Belief States in Multi-Agent Reinforcement Learning
• Auxiliary Tasks Speed Up Learning PointGoal Navigation
• Bridging Imagination and Reality for Model-Based Deep Reinforcement Learning
• Procedural Generalization by Planning with Self-Supervised World Models
• Instance based Generalization in Reinforcement Learning
• There Is No Turning Back: A Self-Supervised Approach for Reversibility-Aware Reinforcement Learning
• No Representation without Transformation
• Robust Robotic Control from Pixels using Contrastive Recurrent State-Space Models
• Pushing it out of the Way: Interactive Visual Navigation
• Temporally Abstract Partial Models