Adaptive Semiparametric Language Models

Abstract

We present a language model that combines a large parametric neural network (i.e., a transformer) with a non-parametric episodic memory component in an integrated architecture. Our model uses extended short-term context by caching local hidden states -- similar to transformer-XL -- and global long-term memory by retrieving a set of nearest neighbor tokens at each timestep. We design a gating function to adaptively combine multiple information sources to make a prediction. This mechanism allows the model to use either local context, short-term memory, or long-term memory (or any combination of them) on an ad hoc basis depending on the context. Experiments on word-based and character-based language modeling datasets demonstrate the efficacy of our proposed method compared to strong baselines.

Accepted to TACL, pre MIT Press publication version

References in corpus (6)

• REALM: Retrieval-Augmented Language Model Pre-Training
• Reformer: The Efficient Transformer
• Learning to Remember Rare Events
• Improving Neural Language Models with a Continuous Cache
• Dynamic Evaluation of Transformer Language Models
• Unbounded cache model for online language modeling with open vocabulary

Cited by in corpus (1)

• Go Forth and Prosper: Language Modeling with Ancient Textual History