Collaborative Distillation for Ultra-Resolution Universal Style Transfer

Abstract

Universal style transfer methods typically leverage rich representations from deep Convolutional Neural Network (CNN) models (e.g., VGG-19) pre-trained on large collections of images. Despite the effectiveness, its application is heavily constrained by the large model size to handle ultra-resolution images given limited memory. In this work, we present a new knowledge distillation method (named Collaborative Distillation) for encoder-decoder based neural style transfer to reduce the convolutional filters. The main idea is underpinned by a finding that the encoder-decoder pairs construct an exclusive collaborative relationship, which is regarded as a new kind of knowledge for style transfer models. Moreover, to overcome the feature size mismatch when applying collaborative distillation, a linear embedding loss is introduced to drive the student network to learn a linear embedding of the teacher's features. Extensive experiments show the effectiveness of our method when applied to different universal style transfer approaches (WCT and AdaIN), even if the model size is reduced by 15.5 times. Especially, on WCT with the compressed models, we achieve ultra-resolution (over 40 megapixels) universal style transfer on a 12GB GPU for the first time. Further experiments on optimization-based stylization scheme show the generality of our algorithm on different stylization paradigms. Our code and trained models are available at https://github.com/mingsun-tse/collaborative-distillation.

arXiv commentAccepted by CVPR 2020, higher-resolution images than the camera-ready version

References in corpus (20)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Distilling the Knowledge in a Neural Network
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• Neural Architecture Search: A Survey
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer
• Do Deep Nets Really Need to be Deep?
• Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
• Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
• A Learned Representation For Artistic Style
• Texture Networks: Feed-forward Synthesis of Textures and Stylized Images
• Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decomposition
• Fast Patch-based Style Transfer of Arbitrary Style
• Stable and Controllable Neural Texture Synthesis and Style Transfer Using Histogram Losses
• Learning Linear Transformations for Fast Arbitrary Style Transfer
• Structured Probabilistic Pruning for Convolutional Neural Network Acceleration
• Accelerating Very Deep Convolutional Networks for Classification and Detection
• Unpaired High-Resolution and Scalable Style Transfer Using Generative Adversarial Networks