Artistic style transfer for videos and spherical images

Abstract

Manually re-drawing an image in a certain artistic style takes a professional artist a long time. Doing this for a video sequence single-handedly is beyond imagination. We present two computational approaches that transfer the style from one image (for example, a painting) to a whole video sequence. In our first approach, we adapt to videos the original image style transfer technique by Gatys et al. based on energy minimization. We introduce new ways of initialization and new loss functions to generate consistent and stable stylized video sequences even in cases with large motion and strong occlusion. Our second approach formulates video stylization as a learning problem. We propose a deep network architecture and training procedures that allow us to stylize arbitrary-length videos in a consistent and stable way, and nearly in real time. We show that the proposed methods clearly outperform simpler baselines both qualitatively and quantitatively. Finally, we propose a way to adapt these approaches also to 360 degree images and videos as they emerge with recent virtual reality hardware.

v3: added ref to conference. This paper is a successor of and overlaps with arXiv:1604.08610, International Journal of Computer Vision (IJCV), 2018

References in corpus (5)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Instance Normalization: The Missing Ingredient for Fast Stylization
• Texture Networks: Feed-forward Synthesis of Textures and Stylized Images
• Artistic style transfer for videos

Cited by in corpus (10)

• Improving data-driven global weather prediction using deep convolutional neural networks on a cubed sphere
• A survey of synthetic data augmentation methods in computer vision
• Sequential vessel segmentation via deep channel attention network
• Automated building image extraction from 360° panoramas for postdisaster evaluation
• Optical Flow Distillation: Towards Efficient and Stable Video Style Transfer
• OSLO: On-the-Sphere Learning for Omnidirectional images and its application to 360-degree image compression
• Evaluation in Neural Style Transfer: A Review
• MM-NeRF: Multimodal-Guided 3D Multi-Style Transfer of Neural Radiance Field
• MVStylizer: An Efficient Edge-Assisted Video Photorealistic Style Transfer System for Mobile Phones
• Stylizing Sparse-View 3D Scenes with Hierarchical Neural Representation