Transferring Inductive Biases through Knowledge Distillation

Abstract

Having the right inductive biases can be crucial in many tasks or scenarios where data or computing resources are a limiting factor, or where training data is not perfectly representative of the conditions at test time. However, defining, designing and efficiently adapting inductive biases is not necessarily straightforward. In this paper, we explore the power of knowledge distillation for transferring the effect of inductive biases from one model to another. We consider families of models with different inductive biases, LSTMs vs. Transformers and CNNs vs. MLPs, in the context of tasks and scenarios where having the right inductive biases is critical. We study the effect of inductive biases on the solutions the models converge to and investigate how and to what extent the effect of inductive biases is transferred through knowledge distillation, in terms of not only performance but also different aspects of converged solutions.

References in corpus (15)

• Distilling the Knowledge in a Neural Network
• Language Models are Few-Shot Learners
• CTRL: A Conditional Transformer Language Model for Controllable Generation
• Universal Transformers
• Distilling Task-Specific Knowledge from BERT into Simple Neural Networks
• Distilling a Neural Network Into a Soft Decision Tree
• Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping
• What is being transferred in transfer learning?
• Improving Multi-Task Deep Neural Networks via Knowledge Distillation for Natural Language Understanding
• Theoretical Limitations of Self-Attention in Neural Sequence Models
• Towards Understanding Knowledge Distillation
• Self-Distillation Amplifies Regularization in Hilbert Space
• Understanding and Improving Knowledge Distillation
• Universality and individuality in neural dynamics across large populations of recurrent networks
• Revisiting Knowledge Distillation via Label Smoothing Regularization

Cited by in corpus (9)

• Self-Distillation Amplifies Regularization in Hilbert Space
• A Survey of Visual Transformers
• The Benchmark Lottery
• Efficient Transformers in Reinforcement Learning using Actor-Learner Distillation
• Even your Teacher Needs Guidance: Ground-Truth Targets Dampen Regularization Imposed by Self-Distillation
• S2SD: Simultaneous Similarity-based Self-Distillation for Deep Metric Learning
• Can RNNs trained on harder subject-verb agreement instances still perform well on easier ones?
• Learn Faster and Forget Slower via Fast and Stable Task Adaptation
• LSTMs Compose (and Learn) Bottom-Up