Efficient Transformers with Dynamic Token Pooling

Abstract

Transformers achieve unrivalled performance in modelling language, but remain inefficient in terms of memory and time complexity. A possible remedy is to reduce the sequence length in the intermediate layers by pooling fixed-length segments of tokens. Nevertheless, natural units of meaning, such as words or phrases, display varying sizes. To address this mismatch, we equip language models with a dynamic-pooling mechanism, which predicts segment boundaries in an autoregressive fashion. We compare several methods to infer boundaries, including end-to-end learning through stochastic re-parameterisation, supervised learning (based on segmentations from subword tokenizers or spikes in conditional entropy), as well as linguistically motivated boundaries. We perform character-level evaluation on texts from multiple datasets and morphologically diverse languages. The results demonstrate that dynamic pooling, which jointly segments and models language, is both faster and more accurate than vanilla Transformers and fixed-length pooling within the same computational budget.

References in corpus (22)

• Representation Learning with Contrastive Predictive Coding
• Categorical Reparameterization with Gumbel-Softmax
• Gaussian Error Linear Units (GELUs)
• Hierarchical Text-Conditional Image Generation with CLIP Latents
• Longformer: The Long-Document Transformer
• Scaling Laws for Neural Language Models
• Emergent Abilities of Large Language Models
• The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
• Linformer: Self-Attention with Linear Complexity
• Adaptive Computation Time for Recurrent Neural Networks
• CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data
• Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets
• Gradient Estimation Using Stochastic Computation Graphs
• CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representation
• Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processing
• Charformer: Fast Character Transformers via Gradient-based Subword Tokenization
• Neural Speed Reading via Skim-RNN
• Fast-Slow Recurrent Neural Networks
• Memorizing Transformers
• Combiner: Full Attention Transformer with Sparse Computation Cost
• Surprisal-Driven Zoneout
• Variable-rate hierarchical CPC leads to acoustic unit discovery in speech

Cited by in corpus (1)

• DARC-CLIP: Dynamic Adaptive Refinement with Cross-Attention for Meme Understanding