Video-to-Video Synthesis

Abstract

We study the problem of video-to-video synthesis, whose goal is to learn a mapping function from an input source video (e.g., a sequence of semantic segmentation masks) to an output photorealistic video that precisely depicts the content of the source video. While its image counterpart, the image-to-image synthesis problem, is a popular topic, the video-to-video synthesis problem is less explored in the literature. Without understanding temporal dynamics, directly applying existing image synthesis approaches to an input video often results in temporally incoherent videos of low visual quality. In this paper, we propose a novel video-to-video synthesis approach under the generative adversarial learning framework. Through carefully-designed generator and discriminator architectures, coupled with a spatio-temporal adversarial objective, we achieve high-resolution, photorealistic, temporally coherent video results on a diverse set of input formats including segmentation masks, sketches, and poses. Experiments on multiple benchmarks show the advantage of our method compared to strong baselines. In particular, our model is capable of synthesizing 2K resolution videos of street scenes up to 30 seconds long, which significantly advances the state-of-the-art of video synthesis. Finally, we apply our approach to future video prediction, outperforming several state-of-the-art competing systems.

arXiv commentIn NeurIPS, 2018. Code, models, and more results are available at https://github.com/NVIDIA/vid2vid

References in corpus (28)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Rethinking Atrous Convolution for Semantic Image Segmentation
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• A Brief Survey of Deep Reinforcement Learning
• Spectral Normalization for Generative Adversarial Networks
• Generative Adversarial Text to Image Synthesis
• Conditional Image Synthesis With Auxiliary Classifier GANs
• Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks
• Improved Training of Wasserstein GANs
• Toward Multimodal Image-to-Image Translation
• The Cityscapes Dataset for Semantic Urban Scene Understanding
• Generating Videos with Scene Dynamics
• Unsupervised Image-to-Image Translation Networks
• Unsupervised Cross-Domain Image Generation
• Decomposing Motion and Content for Natural Video Sequence Prediction
• Deep Predictive Coding Networks for Video Prediction and Unsupervised Learning
• cGANs with Projection Discriminator
• FaceForensics: A Large-scale Video Dataset for Forgery Detection in Human Faces
• Multimodal Unsupervised Image-to-Image Translation
• Unsupervised Learning for Physical Interaction through Video Prediction
• Unsupervised Learning of Disentangled Representations from Video
• Stochastic Adversarial Video Prediction
• Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks
• Detailed Garment Recovery from a Single-View Image
• Coherent Online Video Style Transfer
• Characterizing and Improving Stability in Neural Style Transfer
• Pose Guided Human Video Generation

Cited by in corpus (63)

• Media Forensics and DeepFakes: an overview
• Deep-Learning Inversion of Seismic Data
• A Review on Generative Adversarial Networks: Algorithms, Theory, and Applications
• Unsupervised Discovery of Interpretable Directions in the GAN Latent Space
• VideoGPT: Video Generation using VQ-VAE and Transformers
• Adversarial Video Generation on Complex Datasets
• HiFaceGAN: Face Renovation via Collaborative Suppression and Replenishment
• Train Sparsely, Generate Densely: Memory-efficient Unsupervised Training of High-resolution Temporal GAN
• Learning to dance: A graph convolutional adversarial network to generate realistic dance motions from audio
• On Leveraging Pretrained GANs for Generation with Limited Data
• Dance Revolution: Long-Term Dance Generation with Music via Curriculum Learning
• Yes, we GAN: Applying Adversarial Techniques for Autonomous Driving
• Only Train Once: A One-Shot Neural Network Training And Pruning Framework
• Detecting Photoshopped Faces by Scripting Photoshop
• Deep Image Spatial Transformation for Person Image Generation
• StyleGAN2 Distillation for Feed-forward Image Manipulation
• Vid2Actor: Free-viewpoint Animatable Person Synthesis from Video in the Wild
• Controllable Person Image Synthesis with Attribute-Decomposed GAN
• Human Motion Transfer with 3D Constraints and Detail Enhancement
• Talking Face Generation by Conditional Recurrent Adversarial Network
• COVID-19 CT Image Synthesis with a Conditional Generative Adversarial Network
• Mimicry: Towards the Reproducibility of GAN Research
• Style Generator Inversion for Image Enhancement and Animation
• Rethinking Spatially-Adaptive Normalization
• Optimal Unsupervised Domain Translation
• Latent Translation: Crossing Modalities by Bridging Generative Models
• Video Generation from Single Semantic Label Map
• Free-form Video Inpainting with 3D Gated Convolution and Temporal PatchGAN
• Coordinate-based Texture Inpainting for Pose-Guided Image Generation
• RPGAN: GANs Interpretability via Random Routing
• FLNet: Landmark Driven Fetching and Learning Network for Faithful Talking Facial Animation Synthesis
• Few-shot Video-to-Video Synthesis
• Sim-Real Joint Reinforcement Transfer for 3D Indoor Navigation
• Label-Noise Robust Generative Adversarial Networks
• Mocycle-GAN: Unpaired Video-to-Video Translation
• Semantic Road Layout Understanding by Generative Adversarial Inpainting
• Decentralized Attribution of Generative Models
• A Realistic Face-to-Face Conversation System based on Deep Neural Networks
• Diverse Semantic Image Synthesis via Probability Distribution Modeling
• Compositional Video Synthesis with Action Graphs
• Example-Guided Image Synthesis across Arbitrary Scenes using Masked Spatial-Channel Attention and Self-Supervision
• SurfelGAN: Synthesizing Realistic Sensor Data for Autonomous Driving
• Flow Guided Transformable Bottleneck Networks for Motion Retargeting
• Example-Guided Scene Image Synthesis using Masked Spatial-Channel Attention and Patch-Based Self-Supervision
• GAN Memory with No Forgetting
• Smile Like You Mean It: Driving Animatronic Robotic Face with Learned Models
• Generative Smoke Removal
• Sliced Wasserstein Generative Models
• Deep Blind Video Decaptioning by Temporal Aggregation and Recurrence
• Alternating ConvLSTM: Learning Force Propagation with Alternate State Updates
• Short-Term and Long-Term Context Aggregation Network for Video Inpainting
• Multivariate-Information Adversarial Ensemble for Scalable Joint Distribution Matching
• Sound2Sight: Generating Visual Dynamics from Sound and Context
• VisionNet: A Drivable-space-based Interactive Motion Prediction Network for Autonomous Driving
• Inserting Videos into Videos
• Align-and-Attend Network for Globally and Locally Coherent Video Inpainting
• An Efficient Style Virtual Try on Network for Clothing Business Industry
• Learnable Sampling 3D Convolution for Video Enhancement and Action Recognition
• VHS to HDTV Video Translation using Multi-task Adversarial Learning
• Learning Semantic-Aware Dynamics for Video Prediction
• Generative Adversarial Networks for Video-to-Video Domain Adaptation
• Blurry Video Frame Interpolation
• Cross-View Exocentric to Egocentric Video Synthesis