Video Pixel Networks

Abstract

We propose a probabilistic video model, the Video Pixel Network (VPN), that estimates the discrete joint distribution of the raw pixel values in a video. The model and the neural architecture reflect the time, space and color structure of video tensors and encode it as a four-dimensional dependency chain. The VPN approaches the best possible performance on the Moving MNIST benchmark, a leap over the previous state of the art, and the generated videos show only minor deviations from the ground truth. The VPN also produces detailed samples on the action-conditional Robotic Pushing benchmark and generalizes to the motion of novel objects.

16 pages

Cited by in corpus (13)

• Deep Learning for Physical Processes: Incorporating Prior Scientific Knowledge
• Train Sparsely, Generate Densely: Memory-efficient Unsupervised Training of High-resolution Temporal GAN
• Machine Learning for Spatiotemporal Sequence Forecasting: A Survey
• Scaling Autoregressive Video Models
• Improving Video Generation for Multi-functional Applications
• Reduced-Gate Convolutional LSTM Using Predictive Coding for Spatiotemporal Prediction
• Hierarchical Sequence to Sequence Voice Conversion with Limited Data
• Probabilistic Video Generation using Holistic Attribute Control
• Future Segmentation Using 3D Structure
• Frequency Domain Transformer Networks for Video Prediction
• Image2GIF: Generating Cinemagraphs using Recurrent Deep Q-Networks
• Recurrent Deconvolutional Generative Adversarial Networks with Application to Text Guided Video Generation
• Planning in Dynamic Environments with Conditional Autoregressive Models