Self-Referenced Deep Learning

Abstract

Knowledge distillation is an effective approach to transferring knowledge from a teacher neural network to a student target network for satisfying the low-memory and fast running requirements in practice use. Whilst being able to create stronger target networks compared to the vanilla non-teacher based learning strategy, this scheme needs to train additionally a large teacher model with expensive computational cost. In this work, we present a Self-Referenced Deep Learning (SRDL) strategy. Unlike both vanilla optimisation and existing knowledge distillation, SRDL distils the knowledge discovered by the in-training target model back to itself to regularise the subsequent learning procedure therefore eliminating the need for training a large teacher model. SRDL improves the model generalisation performance compared to vanilla learning and conventional knowledge distillation approaches with negligible extra computational cost. Extensive evaluations show that a variety of deep networks benefit from SRDL resulting in enhanced deployment performance on both coarse-grained object categorisation tasks (CIFAR10, CIFAR100, Tiny ImageNet, and ImageNet) and fine-grained person instance identification tasks (Market-1501).

arXiv commentTo Appear in Asian Conference on Computer Vision 2018

References in corpus (8)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Do Deep Nets Really Need to be Deep?
• Knowledge Distillation by On-the-Fly Native Ensemble
• All you need is a good init
• Data-dependent Initializations of Convolutional Neural Networks
• HydraPlus-Net: Attentive Deep Features for Pedestrian Analysis
• Real-time Action Recognition with Enhanced Motion Vector CNNs
• Person Search by Multi-Scale Matching