Video Imagination from a Single Image with Transformation Generation

Abstract

In this work, we focus on a challenging task: synthesizing multiple imaginary videos given a single image. Major problems come from high dimensionality of pixel space and the ambiguity of potential motions. To overcome those problems, we propose a new framework that produce imaginary videos by transformation generation. The generated transformations are applied to the original image in a novel volumetric merge network to reconstruct frames in imaginary video. Through sampling different latent variables, our method can output different imaginary video samples. The framework is trained in an adversarial way with unsupervised learning. For evaluation, we propose a new assessment metric $RIQA$. In experiments, we test on 3 datasets varying from synthetic data to natural scene. Our framework achieves promising performance in image quality assessment. The visual inspection indicates that it can successfully generate diverse five-frame videos in acceptable perceptual quality.

9 pages, 10 figures

References in corpus (11)

• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks
• DRAW: A Recurrent Neural Network For Image Generation
• Generating Videos with Scene Dynamics
• A note on the evaluation of generative models
• Video (language) modeling: a baseline for generative models of natural videos
• Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks
• Transformation-Based Models of Video Sequences
• Synthesizing Dynamic Patterns by Spatial-Temporal Generative ConvNet

Cited by in corpus (12)

• A Review on Deep Learning Techniques for Video Prediction
• Stochastic Adversarial Video Prediction
• Stochastic Variational Video Prediction
• Future Frame Prediction for Anomaly Detection -- A New Baseline
• From Here to There: Video Inbetweening Using Direct 3D Convolutions
• Unsupervised Bi-directional Flow-based Video Generation from one Snapshot
• Inferring Dynamic Representations of Facial Actions from a Still Image
• Learning Long-term Visual Dynamics with Region Proposal Interaction Networks
• 3D Human motion anticipation and classification
• DeepLandscape: Adversarial Modeling of Landscape Video
• Compositional Video Prediction
• VP-GO: a "light" action-conditioned visual prediction model