ALBERT: A Lite BERT for Self-supervised Learning of Language Representations

Abstract

Increasing model size when pretraining natural language representations often results in improved performance on downstream tasks. However, at some point further model increases become harder due to GPU/TPU memory limitations and longer training times. To address these problems, we present two parameter-reduction techniques to lower memory consumption and increase the training speed of BERT. Comprehensive empirical evidence shows that our proposed methods lead to models that scale much better compared to the original BERT. We also use a self-supervised loss that focuses on modeling inter-sentence coherence, and show it consistently helps downstream tasks with multi-sentence inputs. As a result, our best model establishes new state-of-the-art results on the GLUE, RACE, and \squad benchmarks while having fewer parameters compared to BERT-large. The code and the pretrained models are available at https://github.com/google-research/ALBERT.

References in corpus (19)

• Distributed Representations of Words and Phrases and their Compositionality
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Gaussian Error Linear Units (GELUs)
• XLNet: Generalized Autoregressive Pretraining for Language Understanding
• Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
• Skip-Thought Vectors
• Training Deep Nets with Sublinear Memory Cost
• Generating Long Sequences with Sparse Transformers
• Semi-supervised Sequence Learning
• Well-Read Students Learn Better: On the Importance of Pre-training Compact Models
• Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books
• Deep Equilibrium Models
• The Reversible Residual Network: Backpropagation Without Storing Activations
• Large Batch Optimization for Deep Learning: Training BERT in 76 minutes
• Discourse-Based Objectives for Fast Unsupervised Sentence Representation Learning
• StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understanding
• Bi-Directional Block Self-Attention for Fast and Memory-Efficient Sequence Modeling
• Adaptive Input Representations for Neural Language Modeling
• Dual Co-Matching Network for Multi-choice Reading Comprehension

Cited by in corpus (180)

• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• HuggingFace's Transformers: State-of-the-art Natural Language Processing
• Language Models are Few-Shot Learners
• XLNet: Generalized Autoregressive Pretraining for Language Understanding
• Pre-trained Models for Natural Language Processing: A Survey
• Underspecification Presents Challenges for Credibility in Modern Machine Learning
• ChemBERTa: Large-Scale Self-Supervised Pretraining for Molecular Property Prediction
• Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoders
• A Survey of End-to-End Driving: Architectures and Training Methods
• VLP: A Survey on Vision-Language Pre-training
• Controllable Protein Design with Language Models
• ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generation
• Transformers in Healthcare: A Survey
• Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey
• Cyberbullying Detection for Low-resource Languages and Dialects: Review of the State of the Art
• ConvBERT: Improving BERT with Span-based Dynamic Convolution
• Measuring and Reducing Gendered Correlations in Pre-trained Models
• WT5?! Training Text-to-Text Models to Explain their Predictions
• Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processing
• Open-Retrieval Conversational Question Answering
• Confronting Abusive Language Online: A Survey from the Ethical and Human Rights Perspective
• MisRoBÆRTa: Transformers versus Misinformation
• Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth
• Explainable Identification of Dementia from Transcripts using Transformer Networks
• Clover: Toward Sustainable AI with Carbon-Aware Machine Learning Inference Service
• Recent Advances in Natural Language Inference: A Survey of Benchmarks, Resources, and Approaches
• Med-BERT: pre-trained contextualized embeddings on large-scale structured electronic health records for disease prediction
• Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers
• Remix: Rebalanced Mixup
• FastBERT: a Self-distilling BERT with Adaptive Inference Time
• A Survey on Transfer Learning in Natural Language Processing
• NAS-BERT: Task-Agnostic and Adaptive-Size BERT Compression with Neural Architecture Search
• CLUENER2020: Fine-grained Named Entity Recognition Dataset and Benchmark for Chinese
• Machine Reading Comprehension: The Role of Contextualized Language Models and Beyond
• Explaining Question Answering Models through Text Generation
• Using Prior Knowledge to Guide BERT's Attention in Semantic Textual Matching Tasks
• Modifying Memories in Transformer Models
• MUSER: A MUlti-Step Evidence Retrieval Enhancement Framework for Fake News Detection
• Refiner: Refining Self-attention for Vision Transformers
• Red Alarm for Pre-trained Models: Universal Vulnerability to Neuron-Level Backdoor Attacks
• Analyzing Sustainability Reports Using Natural Language Processing
• A rule-free workflow for the automated generation of databases from scientific literature
• I Know What You Are Searching For: Code Snippet Recommendation from Stack Overflow Posts
• Text-based automatic personality prediction: A bibliographic review
• What does BERT Learn from Multiple-Choice Reading Comprehension Datasets?
• Clinical Insights: A Comprehensive Review of Language Models in Medicine
• Transformer-Based Models for Automatic Identification of Argument Relations: A Cross-Domain Evaluation
• A Comprehensive Review and a Taxonomy of Edge Machine Learning: Requirements, Paradigms, and Techniques
• Federated Few-Shot Learning for Mobile NLP
• A Benchmark for Lease Contract Review
• TRANS-BLSTM: Transformer with Bidirectional LSTM for Language Understanding
• A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
• Fake or Genuine? Contextualised Text Representation for Fake Review Detection
• Learning to Encode Position for Transformer with Continuous Dynamical Model
• Low-Dimensional Structure in the Space of Language Representations is Reflected in Brain Responses
• Safety Analysis in the Era of Large Language Models: A Case Study of STPA using ChatGPT
• Optimal Subarchitecture Extraction For BERT
• Towards Interpretable Natural Language Understanding with Explanations as Latent Variables
• Factual Probing Is [MASK]: Learning vs. Learning to Recall
• Cross-lingual Information Retrieval with BERT
• jiant: A Software Toolkit for Research on General-Purpose Text Understanding Models
• Semantic Networks for Engineering Design: A Survey
• Hostility Detection and Covid-19 Fake News Detection in Social Media
• CATBERT: Context-Aware Tiny BERT for Detecting Social Engineering Emails
• DialBERT: A Hierarchical Pre-Trained Model for Conversation Disentanglement
• Transformer-based Approaches for Legal Text Processing
• Character-level HyperNetworks for Hate Speech Detection
• Vision-Language Navigation with Self-Supervised Auxiliary Reasoning Tasks
• Clinical XLNet: Modeling Sequential Clinical Notes and Predicting Prolonged Mechanical Ventilation
• CDistNet: Perceiving Multi-Domain Character Distance for Robust Text Recognition
• Pre-Trained Models: Past, Present and Future
• Relation Classification as Two-way Span-Prediction
• SUPERT: Towards New Frontiers in Unsupervised Evaluation Metrics for Multi-Document Summarization
• Relative Positional Encoding for Transformers with Linear Complexity
• SBERT-WK: A Sentence Embedding Method by Dissecting BERT-based Word Models
• The Life Cycle of Knowledge in Big Language Models: A Survey
• KDLSQ-BERT: A Quantized Bert Combining Knowledge Distillation with Learned Step Size Quantization
• Classification of Multimodal Hate Speech -- The Winning Solution of Hateful Memes Challenge
• Pre-training Text-to-Text Transformers for Concept-centric Common Sense
• PharmMT: A Neural Machine Translation Approach to Simplify Prescription Directions
• LMSanitator: Defending Prompt-Tuning Against Task-Agnostic Backdoors
• GNN is a Counter? Revisiting GNN for Question Answering
• Towards Preemptive Detection of Depression and Anxiety in Twitter
• Improve Transformer Models with Better Relative Position Embeddings
• PruneNet: Channel Pruning via Global Importance
• MC-BERT: Efficient Language Pre-Training via a Meta Controller
• Alquist 4.0: Towards Social Intelligence Using Generative Models and Dialogue Personalization
• Processing Natural Language on Embedded Devices: How Well Do Transformer Models Perform?
• Progressively Stacking 2.0: A Multi-stage Layerwise Training Method for BERT Training Speedup
• SCARF: Self-Supervised Contrastive Learning using Random Feature Corruption
• End-to-End Trainable Multi-Instance Pose Estimation with Transformers
• Training Multilingual Pre-trained Language Model with Byte-level Subwords
• Bimodal Speech Emotion Recognition Using Pre-Trained Language Models
• ParaLaw Nets -- Cross-lingual Sentence-level Pretraining for Legal Text Processing
• Automated essay scoring using efficient transformer-based language models
• Towards Evaluating the Robustness of Chinese BERT Classifiers
• ExT5: Towards Extreme Multi-Task Scaling for Transfer Learning
• Sketch-BERT: Learning Sketch Bidirectional Encoder Representation from Transformers by Self-supervised Learning of Sketch Gestalt
• E2E-VLP: End-to-End Vision-Language Pre-training Enhanced by Visual Learning
• GNN-XML: Graph Neural Networks for Extreme Multi-label Text Classification
• DeepPatent2: A Large-Scale Benchmarking Corpus for Technical Drawing Understanding
• Alquist 3.0: Alexa Prize Bot Using Conversational Knowledge Graph
• Towards Practical Few-shot Federated NLP
• DziriBERT: a Pre-trained Language Model for the Algerian Dialect
• Language Representation Models for Fine-Grained Sentiment Classification
• Exploring BERT Parameter Efficiency on the Stanford Question Answering Dataset v2.0
• Grid Search Hyperparameter Benchmarking of BERT, ALBERT, and LongFormer on DuoRC
• Hybrid Generative-Contrastive Representation Learning
• S-SimCSE: Sampled Sub-networks for Contrastive Learning of Sentence Embedding
• YANMTT: Yet Another Neural Machine Translation Toolkit
• HyperGrid: Efficient Multi-Task Transformers with Grid-wise Decomposable Hyper Projections
• Pretraining with Contrastive Sentence Objectives Improves Discourse Performance of Language Models
• Train and You'll Miss It: Interactive Model Iteration with Weak Supervision and Pre-Trained Embeddings
• Modeling Document Interactions for Learning to Rank with Regularized Self-Attention
• NanoFlow: Scalable Normalizing Flows with Sublinear Parameter Complexity
• kk2018 at SemEval-2020 Task 9: Adversarial Training for Code-Mixing Sentiment Classification
• On the Distributional Properties of Adaptive Gradients
• MVP-BERT: Redesigning Vocabularies for Chinese BERT and Multi-Vocab Pretraining
• CMV-BERT: Contrastive multi-vocab pretraining of BERT
• Staircase Attention for Recurrent Processing of Sequences
• Key Information Extraction From Documents: Evaluation And Generator
• Is Attention always needed? A Case Study on Language Identification from Speech
• Improving Self-supervised Pre-training via a Fully-Explored Masked Language Model
• ReMP: Rectified Metric Propagation for Few-Shot Learning
• BET: A Backtranslation Approach for Easy Data Augmentation in Transformer-based Paraphrase Identification Context
• Is Attention All What You Need? -- An Empirical Investigation on Convolution-Based Active Memory and Self-Attention
• Optimizing small BERTs trained for German NER
• Low Rank Factorization for Compact Multi-Head Self-Attention
• Alignment Attention by Matching Key and Query Distributions
• Pre-training Protein Language Models with Label-Agnostic Binding Pairs Enhances Performance in Downstream Tasks
• An Experimental Evaluation of Transformer-based Language Models in the Biomedical Domain
• WikiCheck: An end-to-end open source Automatic Fact-Checking API based on Wikipedia
• DSEE: Dually Sparsity-embedded Efficient Tuning of Pre-trained Language Models
• A Brief Summary of Interactions Between Meta-Learning and Self-Supervised Learning
• Multilingual Answer Sentence Reranking via Automatically Translated Data
• A Pairwise Probe for Understanding BERT Fine-Tuning on Machine Reading Comprehension
• Linguistically Informed Masking for Representation Learning in the Patent Domain
• Cocktail: Leveraging Ensemble Learning for Optimized Model Serving in Public Cloud
• EvoText: Enhancing Natural Language Generation Models via Self-Escalation Learning for Up-to-Date Knowledge and Improved Performance
• Deep Learning Brasil -- NLP at SemEval-2020 Task 9: Overview of Sentiment Analysis of Code-Mixed Tweets
• Exceeding the Limits of Visual-Linguistic Multi-Task Learning
• Pre-trained Language Model based Ranking in Baidu Search
• Multiplicative Position-aware Transformer Models for Language Understanding
• Finding Friends and Flipping Frenemies: Automatic Paraphrase Dataset Augmentation Using Graph Theory
• Zero-Shot Open-Book Question Answering
• MNet-Sim: A Multi-layered Semantic Similarity Network to Evaluate Sentence Similarity
• Can Deep Neural Networks Predict Data Correlations from Column Names?
• Towards Confident Machine Reading Comprehension
• Do language models learn typicality judgments from text?
• Learning to Learn to be Right for the Right Reasons
• Bayesian Attention Modules
• Knowledge Distillation for Improved Accuracy in Spoken Question Answering
• Variants of BERT, Random Forests and SVM approach for Multimodal Emotion-Target Sub-challenge
• Benchmarking down-scaled (not so large) pre-trained language models
• A BERT-based Distractor Generation Scheme with Multi-tasking and Negative Answer Training Strategies
• LegoFormer: Transformers for Block-by-Block Multi-view 3D Reconstruction
• VAULT: VAriable Unified Long Text Representation for Machine Reading Comprehension
• Neural Delay Differential Equations
• Careful analysis of XRD patterns with Attention
• The Efficiency Misnomer
• A Transformer Based Pitch Sequence Autoencoder with MIDI Augmentation
• Improving speech recognition models with small samples for air traffic control systems
• BURT: BERT-inspired Universal Representation from Learning Meaningful Segment
• Empirical Study on the Software Engineering Practices in Open Source ML Package Repositories
• Fine-tuning Multi-hop Question Answering with Hierarchical Graph Network
• The Brownian motion in the transformer model
• Data-Driven Self-Supervised Graph Representation Learning
• MUSE2020 challenge report
• UPB at SemEval-2020 Task 12: Multilingual Offensive Language Detection on Social Media by Fine-tuning a Variety of BERT-based Models
• Learning Universal Representations from Word to Sentence
• Syntax Role for Neural Semantic Role Labeling
• A text autoencoder from transformer for fast encoding language representation
• MMIU: Dataset for Visual Intent Understanding in Multimodal Assistants
• Ensemble ALBERT on SQuAD 2.0
• Global Greedy Dependency Parsing
• Injecting Hierarchy with U-Net Transformers
• Ensembling Strategies for Answering Natural Questions
• KaLM at SemEval-2020 Task 4: Knowledge-aware Language Models for Comprehension And Generation
• Normalization of Input-output Shared Embeddings in Text Generation Models
• When to Fold'em: How to answer Unanswerable questions