Deep Variational Luenberger-type Observer for Stochastic Video Prediction

Abstract

Considering the inherent stochasticity and uncertainty, predicting future video frames is exceptionally challenging. In this work, we study the problem of video prediction by combining interpretability of stochastic state space models and representation learning of deep neural networks. Our model builds upon an variational encoder which transforms the input video into a latent feature space and a Luenberger-type observer which captures the dynamic evolution of the latent features. This enables the decomposition of videos into static features and dynamics in an unsupervised manner. By deriving the stability theory of the nonlinear Luenberger-type observer, the hidden states in the feature space become insensitive with respect to the initial values, which improves the robustness of the overall model. Furthermore, the variational lower bound on the data log-likelihood can be derived to obtain the tractable posterior prediction distribution based on the variational principle. Finally, the experiments such as the Bouncing Balls dataset and the Pendulum dataset are provided to demonstrate the proposed model outperforms concurrent works.

arXiv commentrewrite paper

References in corpus (18)

• Auto-Encoding Variational Bayes
• Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting
• Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations
• A Recurrent Latent Variable Model for Sequential Data
• Decomposing Motion and Content for Natural Video Sequence Prediction
• Towards a Definition of Disentangled Representations
• Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control
• A Disentangled Recognition and Nonlinear Dynamics Model for Unsupervised Learning
• Sequential Attend, Infer, Repeat: Generative Modelling of Moving Objects
• Unsupervised Learning of Visual Structure using Predictive Generative Networks
• Spatial Broadcast Decoder: A Simple Architecture for Learning Disentangled Representations in VAEs
• Conditional Flow Variational Autoencoders for Structured Sequence Prediction
• Disentangling Factors of Variation Using Few Labels
• Stochastic Prediction of Multi-Agent Interactions from Partial Observations
• Recurrent Kalman Networks: Factorized Inference in High-Dimensional Deep Feature Spaces
• State Space LSTM Models with Particle MCMC Inference
• Unsupervised Discovery of Parts, Structure, and Dynamics
• Disentangled State Space Representations