Adversarial Framework for Unsupervised Learning of Motion Dynamics in Videos
arXiv:1803.09092
Abstract
Human behavior understanding in videos is a complex, still unsolved problem and requires to accurately model motion at both the local (pixel-wise dense prediction) and global (aggregation of motion cues) levels. Current approaches based on supervised learning require large amounts of annotated data, whose scarce availability is one of the main limiting factors to the development of general solutions. Unsupervised learning can instead leverage the vast amount of videos available on the web and it is a promising solution for overcoming the existing limitations. In this paper, we propose an adversarial GAN-based framework that learns video representations and dynamics through a self-supervision mechanism in order to perform dense and global prediction in videos. Our approach synthesizes videos by 1) factorizing the process into the generation of static visual content and motion, 2) learning a suitable representation of a motion latent space in order to enforce spatio-temporal coherency of object trajectories, and 3) incorporating motion estimation and pixel-wise dense prediction into the training procedure. Self-supervision is enforced by using motion masks produced by the generator, as a co-product of its generation process, to supervise the discriminator network in performing dense prediction. Performance evaluation, carried out on standard benchmarks, shows that our approach is able to learn, in an unsupervised way, both local and global video dynamics. The learned representations, then, support the training of video object segmentation methods with sensibly less (about 50%) annotations, giving performance comparable to the state of the art. Furthermore, the proposed method achieves promising performance in generating realistic videos, outperforming state-of-the-art approaches especially on motion-related metrics.
References in corpus (13)
- GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
- UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
- Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials
- Improved Techniques for Training GANs
- DRAW: A Recurrent Neural Network For Image Generation
- Semi-Supervised Learning with Generative Adversarial Networks
- Action-Conditional Video Prediction using Deep Networks in Atari Games
- Decomposing Motion and Content for Natural Video Sequence Prediction
- Unsupervised Learning for Physical Interaction through Video Prediction
- FusionSeg: Learning to combine motion and appearance for fully automatic segmention of generic objects in videos
- Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks
- Video Prediction with Appearance and Motion Conditions
- Hierarchical Video Generation from Orthogonal Information: Optical Flow and Texture