Coherent Online Video Style Transfer

Abstract

Training a feed-forward network for fast neural style transfer of images is proven to be successful. However, the naive extension to process video frame by frame is prone to producing flickering results. We propose the first end-to-end network for online video style transfer, which generates temporally coherent stylized video sequences in near real-time. Two key ideas include an efficient network by incorporating short-term coherence, and propagating short-term coherence to long-term, which ensures the consistency over larger period of time. Our network can incorporate different image stylization networks. We show that the proposed method clearly outperforms the per-frame baseline both qualitatively and quantitatively. Moreover, it can achieve visually comparable coherence to optimization-based video style transfer, but is three orders of magnitudes faster in runtime.

Corrected typos

References in corpus (10)

• Adam: A Method for Stochastic Optimization
• Perceptual Losses for Real-Time Style Transfer and Super-Resolution
• FlowNet: Learning Optical Flow with Convolutional Networks
• Semantic Style Transfer and Turning Two-Bit Doodles into Fine Artworks
• StyleBank: An Explicit Representation for Neural Image Style Transfer
• Combining Markov Random Fields and Convolutional Neural Networks for Image Synthesis
• Optical Flow Estimation using a Spatial Pyramid Network
• Precomputed Real-Time Texture Synthesis with Markovian Generative Adversarial Networks
• DeepMovie: Using Optical Flow and Deep Neural Networks to Stylize Movies
• Deep Feature Flow for Video Recognition

Cited by in corpus (21)

• Video-to-Video Synthesis
• S3D: Single Shot multi-Span Detector via Fully 3D Convolutional Networks
• Gated Context Aggregation Network for Image Dehazing and Deraining
• Towards Open-Set Identity Preserving Face Synthesis
• Attention-guided Temporally Coherent Video Object Matting
• Stereoscopic Neural Style Transfer
• Stroke Controllable Fast Style Transfer with Adaptive Receptive Fields
• Avatar-Net: Multi-scale Zero-shot Style Transfer by Feature Decoration
• ReCoNet: Real-time Coherent Video Style Transfer Network
• Mocycle-GAN: Unpaired Video-to-Video Translation
• Language-Driven Image Style Transfer
• Neural Abstract Style Transfer for Chinese Traditional Painting
• Neural Stereoscopic Image Style Transfer
• Evolvement Constrained Adversarial Learning for Video Style Transfer
• Real-time Localized Photorealistic Video Style Transfer
• Learning Selfie-Friendly Abstraction from Artistic Style Images
• Image Smoothing via Unsupervised Learning
• Learning Blind Video Temporal Consistency
• Generative Adversarial Networks for Video-to-Video Domain Adaptation
• Artistic Object Recognition by Unsupervised Style Adaptation
• Emerging Applications of Reversible Data Hiding