Charformer: Fast Character Transformers via Gradient-based Subword Tokenization

Abstract

State-of-the-art models in natural language processing rely on separate rigid subword tokenization algorithms, which limit their generalization ability and adaptation to new settings. In this paper, we propose a new model inductive bias that learns a subword tokenization end-to-end as part of the model. To this end, we introduce a soft gradient-based subword tokenization module (GBST) that automatically learns latent subword representations from characters in a data-driven fashion. Concretely, GBST enumerates candidate subword blocks and learns to score them in a position-wise fashion using a block scoring network. We additionally introduce Charformer, a deep Transformer model that integrates GBST and operates on the byte level. Via extensive experiments on English GLUE, multilingual, and noisy text datasets, we show that Charformer outperforms a series of competitive byte-level baselines while generally performing on par and sometimes outperforming subword-based models. Additionally, Charformer is fast, improving the speed of both vanilla byte-level and subword-level Transformers by 28%-100% while maintaining competitive quality. We believe this work paves the way for highly performant token-free models that are trained completely end-to-end.

ICLR 2022 Camera Ready

References in corpus (6)

• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Linformer: Self-Attention with Linear Complexity
• CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representation
• Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processing
• Are Pre-trained Convolutions Better than Pre-trained Transformers?
• GLU Variants Improve Transformer

Cited by in corpus (13)

• On the Opportunities and Risks of Foundation Models
• Perceiver IO: A General Architecture for Structured Inputs & Outputs
• Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP
• Impact of Tokenization on Language Models: An Analysis for Turkish
• Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers
• AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing
• Language Modelling with Pixels
• Efficient Transformers with Dynamic Token Pooling
• Fourier Transformer: Fast Long Range Modeling by Removing Sequence Redundancy with FFT Operator
• Are Character-level Translations Worth the Wait? Comparing ByT5 and mT5 for Machine Translation
• SMILE: Evaluation and Domain Adaptation for Social Media Language Understanding
• The Efficiency Misnomer
• Cheap Learning: Maximising Performance of Language Models for Social Data Science Using Minimal Data