How Context Affects Language Models' Factual Predictions

Abstract

When pre-trained on large unsupervised textual corpora, language models are able to store and retrieve factual knowledge to some extent, making it possible to use them directly for zero-shot cloze-style question answering. However, storing factual knowledge in a fixed number of weights of a language model clearly has limitations. Previous approaches have successfully provided access to information outside the model weights using supervised architectures that combine an information retrieval system with a machine reading component. In this paper, we go a step further and integrate information from a retrieval system with a pre-trained language model in a purely unsupervised way. We report that augmenting pre-trained language models in this way dramatically improves performance and that the resulting system, despite being unsupervised, is competitive with a supervised machine reading baseline. Furthermore, processing query and context with different segment tokens allows BERT to utilize its Next Sentence Prediction pre-trained classifier to determine whether the context is relevant or not, substantially improving BERT's zero-shot cloze-style question-answering performance and making its predictions robust to noisy contexts.

accepted at AKBC 2020

References in corpus (8)

• REALM: Retrieval-Augmented Language Model Pre-Training
• Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
• End-to-End Open-Domain Question Answering with BERTserini
• What do you learn from context? Probing for sentence structure in contextualized word representations
• Unsupervised Question Answering by Cloze Translation
• Simple Applications of BERT for Ad Hoc Document Retrieval
• Language Models as Knowledge Bases?
• Linguistic Analysis of Pretrained Sentence Encoders with Acceptability Judgments

Cited by in corpus (20)

• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
• Autoregressive Entity Retrieval
• No More Fine-Tuning? An Experimental Evaluation of Prompt Tuning in Code Intelligence
• How understanding large language models can inform the use of ChatGPT in physics education
• Language Models are Open Knowledge Graphs
• How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answering
• Advanced Semantics for Commonsense Knowledge Extraction
• Measuring and Improving Consistency in Pretrained Language Models
• Video Understanding as Machine Translation
• Comparing Traditional and LLM-based Search for Image Geolocation
• Improving Pre-trained Language Model Fine-tuning with Noise Stability Regularization
• Large Language Models, scientific knowledge and factuality: A framework to streamline human expert evaluation
• Thinking Aloud: Dynamic Context Generation Improves Zero-Shot Reasoning Performance of GPT-2
• Do PLMs Know and Understand Ontological Knowledge?
• Inside ASCENT: Exploring a Deep Commonsense Knowledge Base and its Usage in Question Answering
• CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP
• Zero-shot Slot Filling with DPR and RAG
• Relational World Knowledge Representation in Contextual Language Models: A Review
• P-Adapters: Robustly Extracting Factual Information from Language Models with Diverse Prompts
• Interpreting Language Models Through Knowledge Graph Extraction