Current Limitations of Language Models: What You Need is Retrieval
arXiv:2009.06857
Abstract
We classify and re-examine some of the current approaches to improve the performance-computes trade-off of language models, including (1) non-causal models (such as masked language models), (2) extension of batch length with efficient attention, (3) recurrence, (4) conditional computation and (5) retrieval. We identify some limitations (1) - (4) suffer from. For example, (1) currently struggles with open-ended text generation with the output loosely constrained by the input as well as performing general textual tasks like GPT-2/3 due to its need for a specific fine-tuning dataset. (2) and (3) do not improve the prediction of the first tokens. Scaling up a model size (e.g. efficiently with (4)) still results in poor performance scaling for some tasks. We argue (5) would resolve many of these limitations, and it can (a) reduce the amount of supervision and (b) efficiently extend the context over the entire training dataset and the entire past of the current sample. We speculate how to modify MARGE to perform unsupervised causal modeling that achieves (b) with the retriever jointly trained.
References in corpus (21)
- Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
- Longformer: The Long-Document Transformer
- XLNet: Generalized Autoregressive Pretraining for Language Understanding
- Scaling Laws for Neural Language Models
- REALM: Retrieval-Augmented Language Model Pre-Training
- Generating Long Sequences with Sparse Transformers
- GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
- Big Bird: Transformers for Longer Sequences
- Synthesizer: Rethinking Self-Attention in Transformer Models
- Theoretical Limitations of Self-Attention in Neural Sequence Models
- Pre-training via Paraphrasing
- Depth-Adaptive Transformer
- Compressive Transformers for Long-Range Sequence Modelling
- Cross-lingual Retrieval for Iterative Self-Supervised Training
- DeLighT: Deep and Light-weight Transformer
- Learning to summarize from human feedback
- Training Large Neural Networks with Constant Memory using a New Execution Algorithm
- Augmenting Transformers with KNN-Based Composite Memory for Dialogue
- Surprisal-Triggered Conditional Computation with Neural Networks
- On-The-Fly Information Retrieval Augmentation for Language Models