Pre-trained Summarization Distillation

Abstract

Recent state-of-the-art approaches to summarization utilize large pre-trained Transformer models. Distilling these models to smaller student models has become critically important for practical use; however there are many different distillation methods proposed by the NLP literature. Recent work on distilling BERT for classification and regression tasks shows strong performance using direct knowledge distillation. Alternatively, machine translation practitioners distill using pseudo-labeling, where a small model is trained on the translations of a larger model. A third, simpler approach is to 'shrink and fine-tune' (SFT), which avoids any explicit distillation by copying parameters to a smaller student model and then fine-tuning. We compare these three approaches for distillation of Pegasus and BART, the current and former state of the art, pre-trained summarization models, and find that SFT outperforms knowledge distillation and pseudo-labeling on the CNN/DailyMail dataset, but under-performs pseudo-labeling on the more abstractive XSUM dataset. PyTorch Code and checkpoints of different sizes are available through Hugging Face transformers here http://tiny.cc/4iy0tz.

References in corpus (9)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Distilling the Knowledge in a Neural Network
• MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers
• Multilingual Denoising Pre-training for Neural Machine Translation
• Reducing Transformer Depth on Demand with Structured Dropout
• TinyBERT: Distilling BERT for Natural Language Understanding
• BERT-of-Theseus: Compressing BERT by Progressive Module Replacing
• Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization
• Noisy Self-Knowledge Distillation for Text Summarization

Cited by in corpus (6)

• Hi, my name is Martha: Using names to measure and mitigate bias in generative dialogue models
• EL-Attention: Memory Efficient Lossless Attention for Generation
• FastSeq: Make Sequence Generation Faster
• NU:BRIEF -- A Privacy-aware Newsletter Personalization Engine for Publishers
• Boosting Summarization with Normalizing Flows and Aggressive Training
• ReadOnce Transformers: Reusable Representations of Text for Transformers