Few-shot Video-to-Video Synthesis
arXiv:1910.12713
Abstract
Video-to-video synthesis (vid2vid) aims at converting an input semantic video, such as videos of human poses or segmentation masks, to an output photorealistic video. While the state-of-the-art of vid2vid has advanced significantly, existing approaches share two major limitations. First, they are data-hungry. Numerous images of a target human subject or a scene are required for training. Second, a learned model has limited generalization capability. A pose-to-human vid2vid model can only synthesize poses of the single person in the training set. It does not generalize to other humans that are not in the training set. To address the limitations, we propose a few-shot vid2vid framework, which learns to synthesize videos of previously unseen subjects or scenes by leveraging few example images of the target at test time. Our model achieves this few-shot generalization capability via a novel network weight generation module utilizing an attention mechanism. We conduct extensive experimental validations with comparisons to strong baselines using several large-scale video datasets including human-dancing videos, talking-head videos, and street-scene videos. The experimental results verify the effectiveness of the proposed framework in addressing the two limitations of existing vid2vid approaches.
arXiv commentIn NeurIPS, 2019
References in corpus (34)
- Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
- Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
- GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
- Large Scale GAN Training for High Fidelity Natural Image Synthesis
- Self-Attention Generative Adversarial Networks
- Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks
- Conditional Image Synthesis With Auxiliary Classifier GANs
- Improved Training of Wasserstein GANs
- Toward Multimodal Image-to-Image Translation
- The Cityscapes Dataset for Semantic Urban Scene Understanding
- Generating Videos with Scene Dynamics
- Unsupervised Image-to-Image Translation Networks
- Unsupervised Cross-Domain Image Generation
- Decomposing Motion and Content for Natural Video Sequence Prediction
- Deep Predictive Coding Networks for Video Prediction and Unsupervised Learning
- cGANs with Projection Discriminator
- FaceForensics: A Large-scale Video Dataset for Forgery Detection in Human Faces
- Multimodal Unsupervised Image-to-Image Translation
- Unsupervised Learning for Physical Interaction through Video Prediction
- Semantic Image Synthesis with Spatially-Adaptive Normalization
- Unsupervised Learning of Disentangled Representations from Video
- Stochastic Adversarial Video Prediction
- Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks
- Video-to-Video Synthesis
- Graph HyperNetworks for Neural Architecture Search
- CGNet: A Light-weight Context Guided Network for Semantic Segmentation
- Few-Shot Unsupervised Image-to-Image Translation
- Video Frame Interpolation via Adaptive Separable Convolution
- Video Frame Interpolation via Adaptive Convolution
- Pixel-Adaptive Convolutional Neural Networks
- Video Generation from Single Semantic Label Map
- Vid2Game: Controllable Characters Extracted from Real-World Videos
- Flow-Grounded Spatial-Temporal Video Prediction from Still Images
- Learning to Take Directions One Step at a Time