KTAN: Knowledge Transfer Adversarial Network

Abstract

To reduce the large computation and storage cost of a deep convolutional neural network, the knowledge distillation based methods have pioneered to transfer the generalization ability of a large (teacher) deep network to a light-weight (student) network. However, these methods mostly focus on transferring the probability distribution of the softmax layer in a teacher network and thus neglect the intermediate representations. In this paper, we propose a knowledge transfer adversarial network to better train a student network. Our technique holistically considers both intermediate representations and probability distributions of a teacher network. To transfer the knowledge of intermediate representations, we set high-level teacher feature maps as a target, toward which the student feature maps are trained. Specifically, we arrange a Teacher-to-Student layer for enabling our framework suitable for various student structures. The intermediate representation helps the student network better understand the transferred generalization as compared to the probability distribution only. Furthermore, we infuse an adversarial learning process by employing a discriminator network, which can fully exploit the spatial correlation of feature maps in training a student network. The experimental results demonstrate that the proposed method can significantly improve the performance of a student network on both image classification and object detection tasks.

8 pages, 2 figures

References in corpus (11)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Distilling the Knowledge in a Neural Network
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Deep Residual Learning for Image Recognition
• Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• FitNets: Hints for Thin Deep Nets
• A Survey of Model Compression and Acceleration for Deep Neural Networks
• Associative Embedding: End-to-End Learning for Joint Detection and Grouping
• Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weights
• Deep Descriptor Transforming for Image Co-Localization

Cited by in corpus (8)

• Knowledge Distillation: A Survey
• Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation
• Feature-map-level Online Adversarial Knowledge Distillation
• Distilling Object Detectors with Task Adaptive Regularization
• Knowledge Squeezed Adversarial Network Compression
• P-KDGAN: Progressive Knowledge Distillation with GANs for One-class Novelty Detection
• MemNet: Memory-Efficiency Guided Neural Architecture Search with Augment-Trim learning
• Be Your Own Best Competitor! Multi-Branched Adversarial Knowledge Transfer