Representation Transfer by Optimal Transport

Abstract

Learning generic representations with deep networks requires massive training samples and significant computer resources. To learn a new specific task, an important issue is to transfer the generic teacher's representation to a student network. In this paper, we propose to use a metric between representations that is based on a functional view of neurons. We use optimal transport to quantify the match between two representations, yielding a distance that embeds some invariances inherent to the representation of deep networks. This distance defines a regularizer promoting the similarity of the student's representation with that of the teacher. Our approach can be used in any learning context where representation transfer is applicable. We experiment here on two standard settings: inductive transfer learning, where the teacher's representation is transferred to a student network of same architecture for a new related task, and knowledge distillation, where the teacher's representation is transferred to a student of simpler architecture for the same task (model compression). Our approach also lends itself to solving new learning problems; we demonstrate this by showing how to directly transfer the teacher's representation to a simpler architecture student for a new related task.

References in corpus (18)

• Distilling the Knowledge in a Neural Network
• Deep Residual Learning for Image Recognition
• Learning Transferable Features with Deep Adaptation Networks
• Deep Domain Confusion: Maximizing for Domain Invariance
• Wide Residual Networks
• Fine-Grained Visual Classification of Aircraft
• Similarity of Neural Network Representations Revisited
• Like What You Like: Knowledge Distill via Neuron Selectivity Transfer
• Explicit Inductive Bias for Transfer Learning with Convolutional Networks
• DELTA: DEep Learning Transfer using Feature Map with Attention for Convolutional Networks
• A Fast Proximal Point Method for Computing Exact Wasserstein Distance
• Contrastive Representation Distillation
• Insights on representational similarity in neural networks with canonical correlation
• Relational Knowledge Distillation
• Towards Understanding Learning Representations: To What Extent Do Different Neural Networks Learn the Same Representation
• Wide-Slice Residual Networks for Food Recognition
• Model Fusion via Optimal Transport
• The State of Knowledge Distillation for Classification