On-The-Fly Information Retrieval Augmentation for Language Models
arXiv:2007.01528
Abstract
Here we experiment with the use of information retrieval as an augmentation for pre-trained language models. The text corpus used in information retrieval can be viewed as form of episodic memory which grows over time. By augmenting GPT 2.0 with information retrieval we achieve a zero shot 15% relative reduction in perplexity on Gigaword corpus without any re-training. We also validate our IR augmentation on an event co-reference task.
arXiv commentACL 2020 NUSE Workshop
References in corpus (9)
- Go for a Walk and Arrive at the Answer: Reasoning Over Paths in Knowledge Bases using Reinforcement Learning
- Neural Natural Language Inference Models Enhanced with External Knowledge
- Contextual LSTM (CLSTM) models for Large scale NLP tasks
- Large Memory Layers with Product Keys
- Scaling Memory-Augmented Neural Networks with Sparse Reads and Writes
- Hierarchical Memory Networks
- Memory Augmented Neural Networks with Wormhole Connections
- PullNet: Open Domain Question Answering with Iterative Retrieval on Knowledge Bases and Text
- Linguistic Knowledge as Memory for Recurrent Neural Networks