Which *BERT? A Survey Organizing Contextualized Encoders
arXiv:2010.00854
Abstract
Pretrained contextualized text encoders are now a staple of the NLP community. We present a survey on language representation learning with the aim of consolidating a series of shared lessons learned across a variety of recent efforts. While significant advancements continue at a rapid pace, we find that enough has now been discovered, in different directions, that we can begin to organize advances according to common themes. Through this organization, we highlight important considerations when interpreting recent contributions and choosing which model to use.
EMNLP 2020
References in corpus (26)
- Distilling the Knowledge in a Neural Network
- MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
- Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
- Longformer: The Long-Document Transformer
- Scaling Laws for Neural Language Models
- CTRL: A Conditional Transformer Language Model for Controllable Generation
- ERNIE: Enhanced Representation through Knowledge Integration
- Predicting Parameters in Deep Learning
- Multilingual Denoising Pre-training for Neural Machine Translation
- MASS: Masked Sequence to Sequence Pre-training for Language Generation
- Generating Long Sequences with Sparse Transformers
- Reformer: The Efficient Transformer
- XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization
- Assessing BERT's Syntactic Abilities
- FreeLB: Enhanced Adversarial Training for Natural Language Understanding
- Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Tasks
- ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehension
- Cross-Lingual Ability of Multilingual BERT: An Empirical Study
- Insertion Transformer: Flexible Sequence Generation via Insertion Operations
- Multilingual Alignment of Contextual Word Representations
- Discourse-Based Objectives for Fast Unsupervised Sentence Representation Learning
- KERMIT: Generative Insertion-Based Modeling for Sequences
- BP-Transformer: Modelling Long-Range Context via Binary Partitioning
- Reweighted Proximal Pruning for Large-Scale Language Representation
- BERT-of-Theseus: Compressing BERT by Progressive Module Replacing
- Contextual Word Representations: A Contextual Introduction