Relational Knowledge Distillation

Abstract

Knowledge distillation aims at transferring knowledge acquired in one model (a teacher) to another model (a student) that is typically smaller. Previous approaches can be expressed as a form of training the student to mimic output activations of individual data examples represented by the teacher. We introduce a novel approach, dubbed relational knowledge distillation (RKD), that transfers mutual relations of data examples instead. For concrete realizations of RKD, we propose distance-wise and angle-wise distillation losses that penalize structural differences in relations. Experiments conducted on different tasks show that the proposed method improves educated student models with a significant margin. In particular for metric learning, it allows students to outperform their teachers' performance, achieving the state of the arts on standard benchmark datasets.

Accepted to CVPR 2019

References in corpus (12)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Distilling the Knowledge in a Neural Network
• Prototypical Networks for Few-shot Learning
• Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results
• Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer
• Do Deep Nets Really Need to be Deep?
• Object-Part Attention Model for Fine-grained Image Classification
• Like What You Like: Knowledge Distill via Neuron Selectivity Transfer
• Data-Free Knowledge Distillation for Deep Neural Networks
• Label Refinery: Improving ImageNet Classification through Label Progression
• Deep Model Compression: Distilling Knowledge from Noisy Teachers
• Moonshine: Distilling with Cheap Convolutions

Cited by in corpus (10)

• Knowledge Distillation: A Survey
• A Comprehensive Study of Class Incremental Learning Algorithms for Visual Tasks
• Hierarchical Self-supervised Augmented Knowledge Distillation
• Distilling Knowledge via Knowledge Review
• Collaborative Distillation in the Parameter and Spectrum Domains for Video Action Recognition
• Towards Oracle Knowledge Distillation with Neural Architecture Search
• Student Network Learning via Evolutionary Knowledge Distillation
• Representation Transfer by Optimal Transport
• A New Training Framework for Deep Neural Network
• Deep Epidemiological Modeling by Black-box Knowledge Distillation: An Accurate Deep Learning Model for COVID-19