Characterizing and Improving Stability in Neural Style Transfer
arXiv:1705.02092
Abstract
Recent progress in style transfer on images has focused on improving the quality of stylized images and speed of methods. However, real-time methods are highly unstable resulting in visible flickering when applied to videos. In this work we characterize the instability of these methods by examining the solution set of the style transfer objective. We show that the trace of the Gram matrix representing style is inversely related to the stability of the method. Then, we present a recurrent convolutional network for real-time video style transfer which incorporates a temporal consistency loss and overcomes the instability of prior methods. Our networks can be applied at any resolution, do not re- quire optical flow at test time, and produce high quality, temporally consistent stylized videos in real-time.
References in corpus (9)
- Adam: A Method for Stochastic Optimization
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Deep Residual Learning for Image Recognition
- A Learned Representation For Artistic Style
- FlowNet: Learning Optical Flow with Convolutional Networks
- Texture Networks: Feed-forward Synthesis of Textures and Stylized Images
- Texture Synthesis Using Convolutional Neural Networks
- Fast Patch-based Style Transfer of Arbitrary Style
- Multimodal Transfer: A Hierarchical Deep Convolutional Neural Network for Fast Artistic Style Transfer
Cited by in corpus (12)
- Video-to-Video Synthesis
- Learning Linear Transformations for Fast Arbitrary Style Transfer
- Stereoscopic Neural Style Transfer
- Avatar-Net: Multi-scale Zero-shot Style Transfer by Feature Decoration
- ReCoNet: Real-time Coherent Video Style Transfer Network
- Mocycle-GAN: Unpaired Video-to-Video Translation
- Language-Driven Image Style Transfer
- Neural Stereoscopic Image Style Transfer
- Recovering Geometric Information with Learned Texture Perturbations
- Learning Blind Video Temporal Consistency
- Temporally Coherent Video Harmonization Using Adversarial Networks
- Video-to-Video Translation for Visual Speech Synthesis