Self-Knowledge Distillation in Natural Language Processing

Abstract

Since deep learning became a key player in natural language processing (NLP), many deep learning models have been showing remarkable performances in a variety of NLP tasks, and in some cases, they are even outperforming humans. Such high performance can be explained by efficient knowledge representation of deep learning models. While many methods have been proposed to learn more efficient representation, knowledge distillation from pretrained deep networks suggest that we can use more information from the soft target probability to train other neural networks. In this paper, we propose a new knowledge distillation method self-knowledge distillation, based on the soft target probabilities of the training model itself, where multimode information is distilled from the word embedding space right below the softmax layer. Due to the time complexity, our method approximates the soft target probabilities. In experiments, we applied the proposed method to two different and fundamental NLP tasks: language model and neural machine translation. The experiment results show that our proposed method improves performance on the tasks.

References in corpus (7)

• Efficient Estimation of Word Representations in Vector Space
• Distilling the Knowledge in a Neural Network
• Sequence to Sequence Learning with Neural Networks
• FitNets: Hints for Thin Deep Nets
• Pointer Sentinel Mixture Models
• Born Again Neural Networks
• A Neural Knowledge Language Model

Cited by in corpus (10)

• Knowledge Distillation: A Survey
• Large-Scale Generative Data-Free Distillation
• Self-Knowledge Distillation with Progressive Refinement of Targets
• Noisy Self-Knowledge Distillation for Text Summarization
• Knowledge Distillation for Multilingual Unsupervised Neural Machine Translation
• Pro-KD: Progressive Distillation by Following the Footsteps of the Teacher
• Learning ULMFiT and Self-Distillation with Calibration for Medical Dialogue System
• Neighbourhood Distillation: On the benefits of non end-to-end distillation
• Learning from Matured Dumb Teacher for Fine Generalization
• Weight Distillation: Transferring the Knowledge in Neural Network Parameters