Fast Vocabulary Transfer for Language Model Compression

Abstract

Real-world business applications require a trade-off between language model performance and size. We propose a new method for model compression that relies on vocabulary transfer. We evaluate the method on various vertical domains and downstream tasks. Our results indicate that vocabulary transfer can be effectively used in combination with other compression techniques, yielding a significant reduction in model size and inference time while marginally compromising on performance.

arXiv commentThe 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP 2022)

References in corpus (6)

• Distilling the Knowledge in a Neural Network
• Introduction to the CoNLL-2002 Shared Task: Language-Independent Named Entity Recognition
• Deep Learning with Limited Numerical Precision
• To prune, or not to prune: exploring the efficacy of pruning for model compression
• MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers
• Fine-Tuning Transformers: Vocabulary Transfer

Cited by in corpus (4)

• Multi-word Tokenization for Sequence Compression
• An energy-based comparative analysis of common approaches to text classification in the Legal domain
• Are Compressed Language Models Less Subgroup Robust?
• Extracting Information in a Low-resource Setting: Case Study on Bioinformatics Workflows