Transferring Inter-Class Correlation
arXiv:2008.10444
Abstract
The Teacher-Student (T-S) framework is widely utilized in the classification tasks, through which the performance of one neural network (the student) can be improved by transferring knowledge from another trained neural network (the teacher). Since the transferring knowledge is related to the network capacities and structures between the teacher and the student, how to define efficient knowledge remains an open question. To address this issue, we design a novel transferring knowledge, the Self-Attention based Inter-Class Correlation (ICC) map in the output layer, and propose our T-S framework, Inter-Class Correlation Transfer (ICCT).
arXiv comment10 pages, 5 figures
References in corpus (6)
- Distilling the Knowledge in a Neural Network
- Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer
- Do Deep Nets Really Need to be Deep?
- A Survey of Model Compression and Acceleration for Deep Neural Networks
- Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
- Using Web Co-occurrence Statistics for Improving Image Categorization