Simple Video Generation using Neural ODEs

Abstract

Despite having been studied to a great extent, the task of conditional generation of sequences of frames, or videos, remains extremely challenging. It is a common belief that a key step towards solving this task resides in modelling accurately both spatial and temporal information in video signals. A promising direction to do so has been to learn latent variable models that predict the future in latent space and project back to pixels, as suggested in recent literature. Following this line of work and building on top of a family of models introduced in prior work, Neural ODE, we investigate an approach that models time-continuous dynamics over a continuous latent space with a differential equation with respect to time. The intuition behind this approach is that these trajectories in latent space could then be extrapolated to generate video frames beyond the time steps for which the model is trained. We show that our approach yields promising results in the task of future frame prediction on the Moving MNIST dataset with 1 and 2 digits.

arXiv comment8 pages, 4 figures, NeurIPS 2019 workshop

References in corpus (8)

• Auto-Encoding Variational Bayes
• Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• The Kinetics Human Action Video Dataset
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• Unsupervised Learning of Video Representations using LSTMs
• A Recurrent Latent Variable Model for Sequential Data
• Stochastic Video Generation with a Learned Prior

Cited by in corpus (3)

• STEER: Simple Temporal Regularization For Neural ODEs
• Neural Ordinary Differential Equations for Data-Driven Reduced Order Modeling of Environmental Hydrodynamics
• STR-GODEs: Spatial-Temporal-Ridership Graph ODEs for Metro Ridership Prediction