Temporal Difference Variational Auto-Encoder

Abstract

To act and plan in complex environments, we posit that agents should have a mental simulator of the world with three characteristics: (a) it should build an abstract state representing the condition of the world; (b) it should form a belief which represents uncertainty on the world; (c) it should go beyond simple step-by-step simulation, and exhibit temporal abstraction. Motivated by the absence of a model satisfying all these requirements, we propose TD-VAE, a generative sequence model that learns representations containing explicit beliefs about states several steps into the future, and that can be rolled out directly without single-step transitions. TD-VAE is trained on pairs of temporally separated time points, using an analogue of temporal difference learning used in reinforcement learning.

References in corpus (22)

• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• WaveNet: A Generative Model for Raw Audio
• Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
• Pixel Recurrent Neural Networks
• A Recurrent Latent Variable Model for Sequential Data
• Semi-Supervised Learning with Ladder Networks
• Value-Function Approximations for Partially Observable Markov Decision Processes
• Professor Forcing: A New Algorithm for Training Recurrent Networks
• A Clockwork RNN
• Hierarchical Multiscale Recurrent Neural Networks
• A Distributional Perspective on Reinforcement Learning
• Stochastic Adversarial Video Prediction
• Learning Stochastic Recurrent Networks
• Deep Kalman Filters
• Recurrent Environment Simulators
• Black box variational inference for state space models
• Learning and Querying Fast Generative Models for Reinforcement Learning
• Generative Temporal Models with Memory
• Time-Agnostic Prediction: Predicting Predictable Video Frames
• Predictive-State Decoders: Encoding the Future into Recurrent Networks
• Learning Awareness Models
• Adaptive Skip Intervals: Temporal Abstraction for Recurrent Dynamical Models

Cited by in corpus (8)

• Learning from Few Samples: A Survey
• Simple and Effective VAE Training with Calibrated Decoders
• Action-Sufficient State Representation Learning for Control with Structural Constraints
• Particle Filter Recurrent Neural Networks
• MELD: Meta-Reinforcement Learning from Images via Latent State Models
• Ensemble Kalman Variational Objectives: Nonlinear Latent Trajectory Inference with A Hybrid of Variational Inference and Ensemble Kalman Filter
• Emergent Communication with World Models
• Variational Predictive Routing with Nested Subjective Timescales