Learning to Take Directions One Step at a Time
arXiv:1812.01874
Abstract
We present a method to generate a video sequence given a single image. Because items in an image can be animated in arbitrarily many different ways, we introduce as control signal a sequence of motion strokes. Such control signal can be automatically transferred from other videos, e.g., via bounding box tracking. Each motion stroke provides the direction to the moving object in the input image and we aim to train a network to generate an animation following a sequence of such directions. To address this task we design a novel recurrent architecture, which can be trained easily and effectively thanks to an explicit separation of past, future and current states. As we demonstrate in the experiments, our proposed architecture is capable of generating an arbitrary number of frames from a single image and a sequence of motion strokes. Key components of our architecture are an autoencoding constraint to ensure consistency with the past and a generative adversarial scheme to ensure that images look realistic and are temporally smooth. We demonstrate the effectiveness of our approach on the MNIST, KTH, Human3.6M, Push and Weizmann datasets.
References in corpus (8)
- Adam: A Method for Stochastic Optimization
- Conditional Generative Adversarial Nets
- GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
- On the difficulty of training Recurrent Neural Networks
- Spectral Normalization for Generative Adversarial Networks
- Conditional Image Generation with PixelCNN Decoders
- Stochastic Video Generation with a Learned Prior
- Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks