A Disentangled Recognition and Nonlinear Dynamics Model for Unsupervised Learning

Abstract

This paper takes a step towards temporal reasoning in a dynamically changing video, not in the pixel space that constitutes its frames, but in a latent space that describes the non-linear dynamics of the objects in its world. We introduce the Kalman variational auto-encoder, a framework for unsupervised learning of sequential data that disentangles two latent representations: an object's representation, coming from a recognition model, and a latent state describing its dynamics. As a result, the evolution of the world can be imagined and missing data imputed, both without the need to generate high dimensional frames at each time step. The model is trained end-to-end on videos of a variety of simulated physical systems, and outperforms competing methods in generative and missing data imputation tasks.

NIPS 2017

References in corpus (13)

• Adam: A Method for Stochastic Optimization
• Auto-Encoding Variational Bayes
• Categorical Reparameterization with Gumbel-Softmax
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• Action-Conditional Video Prediction using Deep Networks in Atari Games
• Unsupervised Learning for Physical Interaction through Video Prediction
• Spatio-temporal video autoencoder with differentiable memory
• Composing graphical models with neural networks for structured representations and fast inference
• A Compositional Object-Based Approach to Learning Physical Dynamics
• Sequential Neural Models with Stochastic Layers
• Recurrent Environment Simulators
• Black box variational inference for state space models
• Backprop KF: Learning Discriminative Deterministic State Estimators

Cited by in corpus (43)

• KalmanNet: Neural Network Aided Kalman Filtering for Partially Known Dynamics
• Discovering physical concepts with neural networks
• Deep Learning for Time Series Forecasting: Tutorial and Literature Survey
• Variational Encoding of Complex Dynamics
• Dynamical Variational Autoencoders: A Comprehensive Review
• Are Disentangled Representations Helpful for Abstract Visual Reasoning?
• Time Series Anomaly Detection for Cyber-Physical Systems via Neural System Identification and Bayesian Filtering
• Weakly-Supervised Disentanglement Without Compromises
• Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Model
• Physics-guided Deep Markov Models for Learning Nonlinear Dynamical Systems with Uncertainty
• Variational Inference of Disentangled Latent Concepts from Unlabeled Observations
• Disentangling Factors of Variation Using Few Labels
• Data-driven Analysis for Understanding Team Sports Behaviors
• IntPhys: A Framework and Benchmark for Visual Intuitive Physics Reasoning
• Learning to Fly via Deep Model-Based Reinforcement Learning
• Interpretable End-to-end Urban Autonomous Driving with Latent Deep Reinforcement Learning
• Deep Factors with Gaussian Processes for Forecasting
• Decentralized policy learning with partial observation and mechanical constraints for multiperson modeling
• Meta-learning autoencoders for few-shot prediction
• Physics-as-Inverse-Graphics: Unsupervised Physical Parameter Estimation from Video
• PDE-Driven Spatiotemporal Disentanglement
• Adaptive Path-Integral Autoencoder: Representation Learning and Planning for Dynamical Systems
• Learning Group Structure and Disentangled Representations of Dynamical Environments
• Modeling Irregular Time Series with Continuous Recurrent Units
• On the Fairness of Disentangled Representations
• DynaNet: Neural Kalman Dynamical Model for Motion Estimation and Prediction
• Tracking by Animation: Unsupervised Learning of Multi-Object Attentive Trackers
• Vid2Param: Modelling of Dynamics Parameters from Video
• Unsupervised Learning of Lagrangian Dynamics from Images for Prediction and Control
• Learning Continuous-Time Dynamics by Stochastic Differential Networks
• Unsupervised dynamic modeling of medical image transformation
• Variational Dynamic Mixtures
• Deep Markov Spatio-Temporal Factorization
• Deep Switching State Space Model (DS${}^3$M) for Nonlinear Time Series Forecasting with Regime Switching
• Ensemble Kalman Variational Objectives: Nonlinear Latent Trajectory Inference with A Hybrid of Variational Inference and Ensemble Kalman Filter
• Re-examination of the Role of Latent Variables in Sequence Modeling
• Equivariant Deep Dynamical Model for Motion Prediction
• On the Impact of Sampling on Deep Sequential State Estimation
• Capturing Actionable Dynamics with Structured Latent Ordinary Differential Equations
• Deep Variational Luenberger-type Observer for Stochastic Video Prediction
• Self-Supervised Inference in State-Space Models
• Switching Recurrent Kalman Networks
• A Benchmark of Dynamical Variational Autoencoders applied to Speech Spectrogram Modeling