Autoregressive Knowledge Distillation through Imitation Learning

Abstract

The performance of autoregressive models on natural language generation tasks has dramatically improved due to the adoption of deep, self-attentive architectures. However, these gains have come at the cost of hindering inference speed, making state-of-the-art models cumbersome to deploy in real-world, time-sensitive settings. We develop a compression technique for autoregressive models that is driven by an imitation learning perspective on knowledge distillation. The algorithm is designed to address the exposure bias problem. On prototypical language generation tasks such as translation and summarization, our method consistently outperforms other distillation algorithms, such as sequence-level knowledge distillation. Student models trained with our method attain 1.4 to 4.8 BLEU/ROUGE points higher than those trained from scratch, while increasing inference speed by up to 14 times in comparison to the teacher model.

References in corpus (24)

• Adam: A Method for Stochastic Optimization
• Distilling the Knowledge in a Neural Network
• Sequence to Sequence Learning with Neural Networks
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
• Unified Language Model Pre-training for Natural Language Understanding and Generation
• Sequence Level Training with Recurrent Neural Networks
• Non-Autoregressive Neural Machine Translation
• An Actor-Critic Algorithm for Sequence Prediction
• How (not) to Train your Generative Model: Scheduled Sampling, Likelihood, Adversary?
• fairseq: A Fast, Extensible Toolkit for Sequence Modeling
• Text Summarization with Pretrained Encoders
• Query-Efficient Imitation Learning for End-to-End Autonomous Driving
• Sequence-to-Sequence Learning as Beam-Search Optimization
• Get To The Point: Summarization with Pointer-Generator Networks
• Sequence-Level Knowledge Distillation
• A Teacher-Student Framework for Zero-Resource Neural Machine Translation
• Learning to Search Better Than Your Teacher
• Latent Alignment and Variational Attention
• Understanding Knowledge Distillation in Non-autoregressive Machine Translation
• Imitation Learning for Non-Autoregressive Neural Machine Translation
• SEARNN: Training RNNs with Global-Local Losses
• Logarithmic Regret for Online Gradient Descent Beyond Strong Convexity