ShrinkTeaNet: Million-scale Lightweight Face Recognition via Shrinking Teacher-Student Networks

Abstract

Large-scale face recognition in-the-wild has been recently achieved matured performance in many real work applications. However, such systems are built on GPU platforms and mostly deploy heavy deep network architectures. Given a high-performance heavy network as a teacher, this work presents a simple and elegant teacher-student learning paradigm, namely ShrinkTeaNet, to train a portable student network that has significantly fewer parameters and competitive accuracy against the teacher network. Far apart from prior teacher-student frameworks mainly focusing on accuracy and compression ratios in closed-set problems, our proposed teacher-student network is proved to be more robust against open-set problem, i.e. large-scale face recognition. In addition, this work introduces a novel Angular Distillation Loss for distilling the feature direction and the sample distributions of the teacher's hypersphere to its student. Then ShrinkTeaNet framework can efficiently guide the student's learning process with the teacher's knowledge presented in both intermediate and last stages of the feature embedding. Evaluations on LFW, CFP-FP, AgeDB, IJB-B and IJB-C Janus, and MegaFace with one million distractors have demonstrated the efficiency of the proposed approach to learn robust student networks which have satisfying accuracy and compact sizes. Our ShrinkTeaNet is able to support the light-weight architecture achieving high performance with 99.77% on LFW and 95.64% on large-scale Megaface protocols.

References in corpus (9)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Distilling the Knowledge in a Neural Network
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Deep Learning Face Representation by Joint Identification-Verification
• Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer
• Do Deep Nets Really Need to be Deep?
• Variational Dropout Sparsifies Deep Neural Networks
• Knowledge Transfer with Jacobian Matching
• Knowledge Transfer via Distillation of Activation Boundaries Formed by Hidden Neurons

Cited by in corpus (7)

• Knowledge Distillation: A Survey
• A Survey on Deep Neural Network Compression: Challenges, Overview, and Solutions
• Joint Feature Distribution Alignment Learning for NIR-VIS and VIS-VIS Face Recognition
• Prototype Memory for Large-scale Face Representation Learning
• VarGFaceNet: An Efficient Variable Group Convolutional Neural Network for Lightweight Face Recognition
• MarginDistillation: distillation for margin-based softmax
• SeesawFaceNets: sparse and robust face verification model for mobile platform