Model Distillation with Knowledge Transfer from Face Classification to Alignment and Verification

Abstract

Knowledge distillation is a potential solution for model compression. The idea is to make a small student network imitate the target of a large teacher network, then the student network can be competitive to the teacher one. Most previous studies focus on model distillation in the classification task, where they propose different architects and initializations for the student network. However, only the classification task is not enough, and other related tasks such as regression and retrieval are barely considered. To solve the problem, in this paper, we take face recognition as a breaking point and propose model distillation with knowledge transfer from face classification to alignment and verification. By selecting appropriate initializations and targets in the knowledge transfer, the distillation can be easier in non-classification tasks. Experiments on the CelebA and CASIA-WebFace datasets demonstrate that the student network can be competitive to the teacher one in alignment and verification, and even surpasses the teacher network under specific compression rates. In addition, to achieve stronger knowledge transfer, we also use a common initialization trick to improve the distillation performance of classification. Evaluations on the CASIA-Webface and large-scale MS-Celeb-1M datasets show the effectiveness of this simple trick.

10 pages, 1 figures

References in corpus (10)

• Distilling the Knowledge in a Neural Network
• Learning Face Representation from Scratch
• Do Deep Nets Really Need to be Deep?
• Compressing Deep Convolutional Networks using Vector Quantization
• An Analysis of Deep Neural Network Models for Practical Applications
• Deep Semantic Ranking Based Hashing for Multi-Label Image Retrieval
• Do Deep Convolutional Nets Really Need to be Deep and Convolutional?
• Deep Model Compression: Distilling Knowledge from Noisy Teachers
• Facial Landmark Detection with Tweaked Convolutional Neural Networks
• How to Train Triplet Networks with 100K Identities?

Cited by in corpus (7)

• Knowledge Distillation: A Survey
• Multitask Emotion Recognition with Incomplete Labels
• On Compressing U-net Using Knowledge Distillation
• Knowledge Distillation as Semiparametric Inference
• Frosting Weights for Better Continual Training
• Distilling Pixel-Wise Feature Similarities for Semantic Segmentation
• CNN inference acceleration using dictionary of centroids