Transformer to CNN: Label-scarce distillation for efficient text classification

Abstract

Significant advances have been made in Natural Language Processing (NLP) modelling since the beginning of 2018. The new approaches allow for accurate results, even when there is little labelled data, because these NLP models can benefit from training on both task-agnostic and task-specific unlabelled data. However, these advantages come with significant size and computational costs. This workshop paper outlines how our proposed convolutional student architecture, having been trained by a distillation process from a large-scale model, can achieve 300x inference speedup and 39x reduction in parameter count. In some cases, the student model performance surpasses its teacher on the studied tasks.

Accepted paper for CDNNRIA workshop at NeurIPS 2018. (3 pages + references)

References in corpus (6)

• Distributed Representations of Words and Phrases and their Compositionality
• Distilling the Knowledge in a Neural Network
• Convolutional Neural Networks for Sentence Classification
• Recurrent Neural Network for Text Classification with Multi-Task Learning
• Pervasive Attention: 2D Convolutional Neural Networks for Sequence-to-Sequence Prediction
• Semi-Supervised Sequence Modeling with Cross-View Training

Cited by in corpus (8)

• Pre-trained Models for Natural Language Processing: A Survey
• Well-Read Students Learn Better: On the Importance of Pre-training Compact Models
• GOBO: Quantizing Attention-Based NLP Models for Low Latency and Energy Efficient Inference
• Adversarial Self-Supervised Data-Free Distillation for Text Classification
• Towards Non-task-specific Distillation of BERT via Sentence Representation Approximation
• Low-Latency Incremental Text-to-Speech Synthesis with Distilled Context Prediction Network
• Distilling Linguistic Context for Language Model Compression
• RefBERT: Compressing BERT by Referencing to Pre-computed Representations