Learning Disentangled Representations of Video with Missing Data
arXiv:2006.13391
Abstract
Missing data poses significant challenges while learning representations of video sequences. We present Disentangled Imputed Video autoEncoder (DIVE), a deep generative model that imputes and predicts future video frames in the presence of missing data. Specifically, DIVE introduces a missingness latent variable, disentangles the hidden video representations into static and dynamic appearance, pose, and missingness factors for each object. DIVE imputes each object's trajectory where data is missing. On a moving MNIST dataset with various missing scenarios, DIVE outperforms the state of the art baselines by a substantial margin. We also present comparisons for real-world MOTSChallenge pedestrian dataset, which demonstrates the practical value of our method in a more realistic setting. Our code and data can be found at https://github.com/Rose-STL-Lab/DIVE.
arXiv commentPublished at NeurIPS 2020
References in corpus (15)
- Unsupervised Learning of Video Representations using LSTMs
- Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
- MOT16: A Benchmark for Multi-Object Tracking
- Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations
- A Recurrent Latent Variable Model for Sequential Data
- GAIN: Missing Data Imputation using Generative Adversarial Nets
- Decomposing Motion and Content for Natural Video Sequence Prediction
- BRITS: Bidirectional Recurrent Imputation for Time Series
- PredRNN++: Towards A Resolution of the Deep-in-Time Dilemma in Spatiotemporal Predictive Learning
- Stochastic Video Generation with a Learned Prior
- Attend, Infer, Repeat: Fast Scene Understanding with Generative Models
- Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks
- Sequential Attend, Infer, Repeat: Generative Modelling of Moving Objects
- Learning Plannable Representations with Causal InfoGAN
- R-SQAIR: Relational Sequential Attend, Infer, Repeat