Scaling Autoregressive Video Models

Abstract

Due to the statistical complexity of video, the high degree of inherent stochasticity, and the sheer amount of data, generating natural video remains a challenging task. State-of-the-art video generation models often attempt to address these issues by combining sometimes complex, usually video-specific neural network architectures, latent variable models, adversarial training and a range of other methods. Despite their often high complexity, these approaches still fall short of generating high quality video continuations outside of narrow domains and often struggle with fidelity. In contrast, we show that conceptually simple autoregressive video generation models based on a three-dimensional self-attention mechanism achieve competitive results across multiple metrics on popular benchmark datasets, for which they produce continuations of high fidelity and realism. We also present results from training our models on Kinetics, a large scale action recognition dataset comprised of YouTube videos exhibiting phenomena such as camera movement, complex object interactions and diverse human movement. While modeling these phenomena consistently remains elusive, we hope that our results, which include occasional realistic continuations encourage further research on comparatively complex, large scale datasets such as Kinetics.

arXiv commentInternational Conference on Learning Representations (ICLR) 2020

References in corpus (17)

• Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Conditional Image Generation with PixelCNN Decoders
• Unsupervised Learning of Video Representations using LSTMs
• Pixel Recurrent Neural Networks
• The Unreasonable Effectiveness of Deep Features as a Perceptual Metric
• Generating Long Sequences with Sparse Transformers
• Model-Based Reinforcement Learning for Atari
• Stochastic Video Generation with a Learned Prior
• A Short Note about Kinetics-600
• Spatio-temporal video autoencoder with differentiable memory
• Image Transformer
• Towards Accurate Generative Models of Video: A New Metric & Challenges
• Generating High Fidelity Images with Subscale Pixel Networks and Multidimensional Upscaling
• Hierarchical Autoregressive Image Models with Auxiliary Decoders
• Video Frame Synthesis using Deep Voxel Flow
• Video Pixel Networks

Cited by in corpus (7)

• Axial Attention in Multidimensional Transformers
• VideoGPT: Video Generation using VQ-VAE and Transformers
• Adversarial Video Generation on Complex Datasets
• Convolutional Tensor-Train LSTM for Spatio-temporal Learning
• RoboNet: Large-Scale Multi-Robot Learning
• Pedestrian Wind Factor Estimation in Complex Urban Environments
• Models, Pixels, and Rewards: Evaluating Design Trade-offs in Visual Model-Based Reinforcement Learning