Adaptive Semiparametric Language Models
arXiv:2102.02557
Abstract
We present a language model that combines a large parametric neural network (i.e., a transformer) with a non-parametric episodic memory component in an integrated architecture. Our model uses extended short-term context by caching local hidden states -- similar to transformer-XL -- and global long-term memory by retrieving a set of nearest neighbor tokens at each timestep. We design a gating function to adaptively combine multiple information sources to make a prediction. This mechanism allows the model to use either local context, short-term memory, or long-term memory (or any combination of them) on an ad hoc basis depending on the context. Experiments on word-based and character-based language modeling datasets demonstrate the efficacy of our proposed method compared to strong baselines.
Accepted to TACL, pre MIT Press publication version
References in corpus (9)
- Longformer: The Long-Document Transformer
- Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
- REALM: Retrieval-Augmented Language Model Pre-Training
- Reformer: The Efficient Transformer
- Learning to Remember Rare Events
- Language Models as Knowledge Bases?
- Improving Neural Language Models with a Continuous Cache
- Dynamic Evaluation of Transformer Language Models
- Unbounded cache model for online language modeling with open vocabulary