Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processing

Abstract

With the success of language pretraining, it is highly desirable to develop more efficient architectures of good scalability that can exploit the abundant unlabeled data at a lower cost. To improve the efficiency, we examine the much-overlooked redundancy in maintaining a full-length token-level presentation, especially for tasks that only require a single-vector presentation of the sequence. With this intuition, we propose Funnel-Transformer which gradually compresses the sequence of hidden states to a shorter one and hence reduces the computation cost. More importantly, by re-investing the saved FLOPs from length reduction in constructing a deeper or wider model, we further improve the model capacity. In addition, to perform token-level predictions as required by common pretraining objectives, Funnel-Transformer is able to recover a deep representation for each token from the reduced hidden sequence via a decoder. Empirically, with comparable or fewer FLOPs, Funnel-Transformer outperforms the standard Transformer on a wide variety of sequence-level prediction tasks, including text classification, language understanding, and reading comprehension. The code and pretrained checkpoints are available at https://github.com/laiguokun/Funnel-Transformer.

References in corpus (7)

• ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
• Pay Less Attention with Lightweight and Dynamic Convolutions
• The Evolved Transformer
• A Mutual Information Maximization Perspective of Language Representation Learning
• HIBERT: Document Level Pre-training of Hierarchical Bidirectional Transformers for Document Summarization
• AdaBERT: Task-Adaptive BERT Compression with Differentiable Neural Architecture Search
• Multiresolution Transformer Networks: Recurrence is Not Essential for Modeling Hierarchical Structure

Cited by in corpus (17)

• CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representation
• Charformer: Fast Character Transformers via Gradient-based Subword Tokenization
• Enhancing Phenotype Recognition in Clinical Notes Using Large Language Models: PhenoBCBERT and PhenoGPT
• Rethinking Spatial Dimensions of Vision Transformers
• Efficient Transformers with Dynamic Token Pooling
• Exploiting Temporal Contexts with Strided Transformer for 3D Human Pose Estimation
• Fourier Transformer: Fast Long Range Modeling by Removing Sequence Redundancy with FFT Operator
• Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Search
• ERNIE-Gram: Pre-Training with Explicitly N-Gram Masked Language Modeling for Natural Language Understanding
• Learning Robust Correlation with Foundation Model for Weakly-Supervised Few-Shot Segmentation
• Condenser: a Pre-training Architecture for Dense Retrieval
• On the Transformer Growth for Progressive BERT Training
• A Simple and Effective Positional Encoding for Transformers
• Dynamic-TinyBERT: Boost TinyBERT's Inference Efficiency by Dynamic Sequence Length
• How could Neural Networks understand Programs?
• PEFA: Parameter-Free Adapters for Large-scale Embedding-based Retrieval Models
• Focus on the Core: Efficient Attention via Pruned Token Compression for Document Classification