Generating Videos with Scene Dynamics

Abstract

We capitalize on large amounts of unlabeled video in order to learn a model of scene dynamics for both video recognition tasks (e.g. action classification) and video generation tasks (e.g. future prediction). We propose a generative adversarial network for video with a spatio-temporal convolutional architecture that untangles the scene's foreground from the background. Experiments suggest this model can generate tiny videos up to a second at full frame rate better than simple baselines, and we show its utility at predicting plausible futures of static images. Moreover, experiments and visualizations show the model internally learns useful features for recognizing actions with minimal supervision, suggesting scene dynamics are a promising signal for representation learning. We believe generative video models can impact many applications in video understanding and simulation.

NIPS 2016. See more at http://web.mit.edu/vondrick/tinyvideo/

References in corpus (5)

• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks
• YFCC100M: The New Data in Multimedia Research
• Deep Predictive Coding Networks for Video Prediction and Unsupervised Learning
• Unsupervised Learning of Visual Representations using Videos

Cited by in corpus (153)

• Deep Learning for Precipitation Nowcasting: A Benchmark and A New Model
• How Generative Adversarial Networks and Their Variants Work: An Overview
• A Review on Deep Learning Techniques for Video Prediction
• Revisiting Unreasonable Effectiveness of Data in Deep Learning Era
• A Review on Generative Adversarial Networks: Algorithms, Theory, and Applications
• Video Coding for Machines: A Paradigm of Collaborative Compression and Intelligent Analytics
• Unsupervised Learning for Physical Interaction through Video Prediction
• Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control
• Learning Video Representations using Contrastive Bidirectional Transformer
• Stochastic Adversarial Video Prediction
• MD-GAN: Multi-Discriminator Generative Adversarial Networks for Distributed Datasets
• DRIT++: Diverse Image-to-Image Translation via Disentangled Representations
• An Introduction to Image Synthesis with Generative Adversarial Nets
• Self-Supervised Learning for Videos: A Survey
• Variational Discriminator Bottleneck: Improving Imitation Learning, Inverse RL, and GANs by Constraining Information Flow
• Generative Adversarial Networks for Spatio-temporal Data: A Survey
• Video-to-Video Synthesis
• Learning to Generate Images of Outdoor Scenes from Attributes and Semantic Layouts
• Train Sparsely, Generate Densely: Memory-efficient Unsupervised Training of High-resolution Temporal GAN
• Self-Supervised Visual Planning with Temporal Skip Connections
• Learning to Decompose and Disentangle Representations for Video Prediction
• Generative Models for Automatic Chemical Design
• Adversarial Attacks Against Deep Generative Models on Data: A Survey
• Photographic Image Synthesis with Cascaded Refinement Networks
• Machine Learning for Spatiotemporal Sequence Forecasting: A Survey
• GODIVA: Generating Open-DomaIn Videos from nAtural Descriptions
• Lat-Net: Compressing Lattice Boltzmann Flow Simulations using Deep Neural Networks
• Dynamic Facial Expression Generation on Hilbert Hypersphere with Conditional Wasserstein Generative Adversarial Nets
• A review of radar-based nowcasting of precipitation and applicable machine learning techniques
• Transformation-Based Models of Video Sequences
• Improving brain computer interface performance by data augmentation with conditional Deep Convolutional Generative Adversarial Networks
• Yes, we GAN: Applying Adversarial Techniques for Autonomous Driving
• Interpreting Deep Visual Representations via Network Dissection
• Generative Models for Low-Rank Video Representation and Reconstruction
• ST-GAN: Spatial Transformer Generative Adversarial Networks for Image Compositing
• Predictive-Corrective Networks for Action Detection
• End-to-End Speech-Driven Facial Animation with Temporal GANs
• COT-GAN: Generating Sequential Data via Causal Optimal Transport
• Transformation-Grounded Image Generation Network for Novel 3D View Synthesis
• Generative Adversarial Networks for Electronic Health Records: A Framework for Exploring and Evaluating Methods for Predicting Drug-Induced Laboratory Test Trajectories
• Lower Dimensional Kernels for Video Discriminators
• Temporal Generative Adversarial Nets with Singular Value Clipping
• Predicting Deeper into the Future of Semantic Segmentation
• Visual Dynamics: Stochastic Future Generation via Layered Cross Convolutional Networks
• 3D Human Pose Estimation in the Wild by Adversarial Learning
• RGB-D-based Human Motion Recognition with Deep Learning: A Survey
• Conditional Adversarial Network for Semantic Segmentation of Brain Tumor
• Cycle-Contrast for Self-Supervised Video Representation Learning
• Unsupervised Learning of Long-Term Motion Dynamics for Videos
• FitVid: Overfitting in Pixel-Level Video Prediction
• Arbitrary Talking Face Generation via Attentional Audio-Visual Coherence Learning
• Self-Supervised Video Representation Learning with Space-Time Cubic Puzzles
• Unsupervised Representation Learning by Sorting Sequences
• Video Imagination from a Single Image with Transformation Generation
• Predicting Video with VQVAE
• Improving colonoscopy lesion classification using semi-supervised deep learning
• Video-based Human Action Recognition using Deep Learning: A Review
• Time-Agnostic Prediction: Predicting Predictable Video Frames
• DiscrimNet: Semi-Supervised Action Recognition from Videos using Generative Adversarial Networks
• Generative Adversarial Networks: A Survey Towards Private and Secure Applications
• Learning to Sketch with Deep Q Networks and Demonstrated Strokes
• GANzilla: User-Driven Direction Discovery in Generative Adversarial Networks
• Video Compression through Image Interpolation
• Talking Face Generation by Conditional Recurrent Adversarial Network
• Message Passing Multi-Agent GANs
• Unsupervised Learning of Video Representations via Dense Trajectory Clustering
• Every Smile is Unique: Landmark-Guided Diverse Smile Generation
• Dual Motion GAN for Future-Flow Embedded Video Prediction
• Packet-Level Adversarial Network Traffic Crafting using Sequence Generative Adversarial Networks
• Accurate and Clear Precipitation Nowcasting with Consecutive Attention and Rain-map Discrimination
• Dynamics Transfer GAN: Generating Video by Transferring Arbitrary Temporal Dynamics from a Source Video to a Single Target Image
• Unsupervised Video-to-Video Translation
• Learning to Generate Time-Lapse Videos Using Multi-Stage Dynamic Generative Adversarial Networks
• Brainstorming Generative Adversarial Networks (BGANs): Towards Multi-Agent Generative Models with Distributed Private Datasets
• Modeling Multiple Views via Implicitly Preserving Global Consistency and Local Complementarity
• Novel Video Prediction for Large-scale Scene using Optical Flow
• To Create What You Tell: Generating Videos from Captions
• Improving Video Generation for Multi-functional Applications
• Motion Prediction Under Multimodality with Conditional Stochastic Networks
• Skip-Clip: Self-Supervised Spatiotemporal Representation Learning by Future Clip Order Ranking
• Distributional Adversarial Networks
• Learning Bidirectional LSTM Networks for Synthesizing 3D Mesh Animation Sequences
• Self-supervised Learning from a Multi-view Perspective
• Flow-Grounded Spatial-Temporal Video Prediction from Still Images
• Disentangling Propagation and Generation for Video Prediction
• MotionVideoGAN: A Novel Video Generator Based on the Motion Space Learned from Image Pairs
• Deep Online Video Stabilization
• Im2Flow: Motion Hallucination from Static Images for Action Recognition
• Generating Neural Networks with Neural Networks
• Vid2Game: Controllable Characters Extracted from Real-World Videos
• Pose Guided Human Video Generation
• Deep Generative Modeling-based Data Augmentation with Demonstration using the BFBT Benchmark Void Fraction Datasets
• The Neural Painter: Multi-Turn Image Generation
• Synthesizing Dynamic Patterns by Spatial-Temporal Generative ConvNet
• ActionFlowNet: Learning Motion Representation for Action Recognition
• 3D-PhysNet: Learning the Intuitive Physics of Non-Rigid Object Deformations
• Facial Expression Video Generation Based-On Spatio-temporal Convolutional GAN: FEV-GAN
• DeepClimGAN: A High-Resolution Climate Data Generator
• Few-shot Video-to-Video Synthesis
• Generative Melody Composition with Human-in-the-Loop Bayesian Optimization
• Learning to Forecast and Refine Residual Motion for Image-to-Video Generation
• Face Translation between Images and Videos using Identity-aware CycleGAN
• Self-Supervised Relative Depth Learning for Urban Scene Understanding
• Planning Robot Motion using Deep Visual Prediction
• Pedestrian Wind Factor Estimation in Complex Urban Environments
• Relational Action Forecasting
• On The State of Data In Computer Vision: Human Annotations Remain Indispensable for Developing Deep Learning Models
• Visual Forecasting by Imitating Dynamics in Natural Sequences
• Learning Dynamic Generator Model by Alternating Back-Propagation Through Time
• Adversarial Video Compression Guided by Soft Edge Detection
• A Neurally-Inspired Hierarchical Prediction Network for Spatiotemporal Sequence Learning and Prediction
• Predictive Learning: Using Future Representation Learning Variantial Autoencoder for Human Action Prediction
• Hybrid VAE: Improving Deep Generative Models using Partial Observations
• Video Jigsaw: Unsupervised Learning of Spatiotemporal Context for Video Action Recognition
• Deep Learning for Low-Dose CT Denoising
• FDNet: A Deep Learning Approach with Two Parallel Cross Encoding Pathways for Precipitation Nowcasting
• C4Synth: Cross-Caption Cycle-Consistent Text-to-Image Synthesis
• Learning Continuous-Time Dynamics by Stochastic Differential Networks
• Zoom-In-to-Check: Boosting Video Interpolation via Instance-level Discrimination
• Conditional Video Generation Using Action-Appearance Captions
• Lip Movements Generation at a Glance
• Image2GIF: Generating Cinemagraphs using Recurrent Deep Q-Networks
• Disentangling Motion, Foreground and Background Features in Videos
• ContextVP: Fully Context-Aware Video Prediction
• Off the Beaten Track: Using Deep Learning to Interpolate Between Music Genres
• Diverse Video Generation using a Gaussian Process Trigger
• Optimal 1-Wasserstein Distance for WGANs
• Evolvement Constrained Adversarial Learning for Video Style Transfer
• Learning Segmentation Masks with the Independence Prior
• Time-Aware and View-Aware Video Rendering for Unsupervised Representation Learning
• Sliced Wasserstein Generative Models
• Hierarchical Model for Long-term Video Prediction
• Human Pose Forecasting via Deep Markov Models
• G3AN: Disentangling Appearance and Motion for Video Generation
• Visual Data Augmentation through Learning
• A Distributed Training Algorithm of Generative Adversarial Networks with Quantized Gradients
• One-Step Time-Dependent Future Video Frame Prediction with a Convolutional Encoder-Decoder Neural Network
• Predicting the Future with Transformational States
• Interpretable Intuitive Physics Model
• Classification of sparsely labeled spatio-temporal data through semi-supervised adversarial learning
• Defo-Net: Learning Body Deformation using Generative Adversarial Networks
• Assisting Scene Graph Generation with Self-Supervision
• Recurrent Flow-Guided Semantic Forecasting
• MT-VAE: Learning Motion Transformations to Generate Multimodal Human Dynamics
• An Interpretable Generative Model for Handwritten Digit Image Synthesis
• Automatic Realistic Music Video Generation from Segments of Youtube Videos
• PixelTransformer: Sample Conditioned Signal Generation
• Self-Supervised Equivariant Scene Synthesis from Video
• Robust One Shot Audio to Video Generation
• Concept Formation and Dynamics of Repeated Inference in Deep Generative Models
• Dynamic Variational Autoencoders for Visual Process Modeling
• cvpaper.challenge in 2016: Futuristic Computer Vision through 1,600 Papers Survey
• Selective Sampling and Mixture Models in Generative Adversarial Networks