Multi-word Tokenization for Sequence Compression

Abstract

Large Language Models have proven highly successful at modelling a variety of tasks. However, this comes at a steep computational cost that hinders wider industrial uptake. In this paper, we present MWT: a Multi-Word Tokenizer that goes beyond word boundaries by representing frequent multi-word expressions as single tokens. MWTs produce a more compact and efficient tokenization that yields two benefits: (1) Increase in performance due to a greater coverage of input data given a fixed sequence length budget; (2) Faster and lighter inference due to the ability to reduce the sequence length with negligible drops in performance. Our results show that MWT is more robust across shorter sequence lengths, thus allowing for major speedups via early sequence truncation.

arXiv commentThe 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP 2023)

References in corpus (10)

• Distilling the Knowledge in a Neural Network
• LLaMA: Open and Efficient Foundation Language Models
• Deep Learning with Limited Numerical Precision
• To prune, or not to prune: exploring the efficacy of pruning for model compression
• MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers
• FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
• Phrase-Based & Neural Unsupervised Machine Translation
• Language Model Tokenizers Introduce Unfairness Between Languages
• Learning to Compress Prompts with Gist Tokens
• Fast Vocabulary Transfer for Language Model Compression