Paraphrasing Complex Network: Network Compression via Factor Transfer

Abstract

Many researchers have sought ways of model compression to reduce the size of a deep neural network (DNN) with minimal performance degradation in order to use DNNs in embedded systems. Among the model compression methods, a method called knowledge transfer is to train a student network with a stronger teacher network. In this paper, we propose a novel knowledge transfer method which uses convolutional operations to paraphrase teacher's knowledge and to translate it for the student. This is done by two convolutional modules, which are called a paraphraser and a translator. The paraphraser is trained in an unsupervised manner to extract the teacher factors which are defined as paraphrased information of the teacher network. The translator located at the student network extracts the student factors and helps to translate the teacher factors by mimicking them. We observed that our student network trained with the proposed factor transfer method outperforms the ones trained with conventional knowledge transfer methods.

Advances in Neural Information Processing Systems

References in corpus (10)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Distilling the Knowledge in a Neural Network
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
• Neural Architecture Search with Reinforcement Learning
• How transferable are features in deep neural networks?
• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• FitNets: Hints for Thin Deep Nets
• Wide Residual Networks
• Energy-based Generative Adversarial Network

Cited by in corpus (14)

• Knowledge Distillation: A Survey
• ResKD: Residual-Guided Knowledge Distillation
• Low-Resolution Object Recognition with Cross-Resolution Relational Contrastive Distillation
• FEED: Feature-level Ensemble for Knowledge Distillation
• Knowledge Adaptation for Efficient Semantic Segmentation
• Zero-Shot Knowledge Distillation from a Decision-Based Black-Box Model
• Distilling Knowledge via Knowledge Review
• Full-Cycle Energy Consumption Benchmark for Low-Carbon Computer Vision
• Edge Bias in Federated Learning and its Solution by Buffered Knowledge Distillation
• Edge-Cloud Collaborated Object Detection via Difficult-Case Discriminator
• On the Orthogonality of Knowledge Distillation with Other Techniques: From an Ensemble Perspective
• CoCo DistillNet: a Cross-layer Correlation Distillation Network for Pathological Gastric Cancer Segmentation
• Student Helping Teacher: Teacher Evolution via Self-Knowledge Distillation
• Multi-granularity for knowledge distillation