Latent Neural Differential Equations for Video Generation

Abstract

Generative Adversarial Networks have recently shown promise for video generation, building off of the success of image generation while also addressing a new challenge: time. Although time was analyzed in some early work, the literature has not adequately grown with temporal modeling developments. We study the effects of Neural Differential Equations to model the temporal dynamics of video generation. The paradigm of Neural Differential Equations presents many theoretical strengths including the first continuous representation of time within video generation. In order to address the effects of Neural Differential Equations, we investigate how changes in temporal models affect generated video quality. Our results give support to the usage of Neural Differential Equations as a simple replacement for older temporal generators. While keeping run times similar and decreasing parameter count, we produce a new state-of-the-art model in 64$\times$64 pixel unconditional video generation, with an Inception Score of 15.20.

References in corpus (19)

• Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Spectral Normalization for Generative Adversarial Networks
• Unsupervised Learning of Video Representations using LSTMs
• Stochastic Video Generation with a Learned Prior
• Video (language) modeling: a baseline for generative models of natural videos
• Stochastic Adversarial Video Prediction
• Adversarial Video Generation on Complex Datasets
• Scalable Gradients for Stochastic Differential Equations
• Train Sparsely, Generate Densely: Memory-efficient Unsupervised Training of High-resolution Temporal GAN
• ODE${}^2$VAE: Deep generative second order ODEs with Bayesian neural networks
• Stochastic Latent Residual Video Prediction
• Neural Stochastic Differential Equations: Deep Latent Gaussian Models in the Diffusion Limit
• Lower Dimensional Kernels for Video Discriminators
• Transformation-based Adversarial Video Prediction on Large-Scale Data
• How does Lipschitz Regularization Influence GAN Training?