Language Models as Knowledge Bases?
arXiv:1909.01066
Abstract
Recent progress in pretraining language models on large textual corpora led to a surge of improvements for downstream NLP tasks. Whilst learning linguistic knowledge, these models may also be storing relational knowledge present in the training data, and may be able to answer queries structured as "fill-in-the-blank" cloze statements. Language models have many advantages over structured knowledge bases: they require no schema engineering, allow practitioners to query about an open class of relations, are easy to extend to more data, and require no human supervision to train. We present an in-depth analysis of the relational knowledge already present (without fine-tuning) in a wide range of state-of-the-art pretrained language models. We find that (i) without fine-tuning, BERT contains relational knowledge competitive with traditional NLP methods that have some access to oracle knowledge, (ii) BERT also does remarkably well on open-domain question answering against a supervised baseline, and (iii) certain types of factual knowledge are learned much more readily than others by standard language model pretraining approaches. The surprisingly strong ability of these models to recall factual knowledge without any fine-tuning demonstrates their potential as unsupervised open-domain QA systems. The code to reproduce our analysis is available at https://github.com/facebookresearch/LAMA.
arXiv commentaccepted at EMNLP 2019
References in corpus (3)
Cited by in corpus (68)
- Learning to Prompt for Vision-Language Models
- REALM: Retrieval-Augmented Language Model Pre-Training
- KnowPrompt: Knowledge-aware Prompt-tuning with Synergistic Optimization for Relation Extraction
- Extracting Training Data from Large Language Models
- CodeBERT: A Pre-Trained Model for Programming and Natural Languages
- Retrieving and Reading: A Comprehensive Survey on Open-domain Question Answering
- Making Pre-trained Language Models Better Few-shot Learners
- Pretrained Transformers for Text Ranking: BERT and Beyond
- Language Models are Open Knowledge Graphs
- How Context Affects Language Models' Factual Predictions
- Large Language Models Can Be Strong Differentially Private Learners
- Calibrate Before Use: Improving Few-Shot Performance of Language Models
- AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts
- QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answering
- Towards Zero-Label Language Learning
- Explaining Question Answering Models through Text Generation
- Modifying Memories in Transformer Models
- Towards Continual Knowledge Learning of Language Models
- Pre-trained Language Models in Biomedical Domain: A Systematic Survey
- Misinformation Has High Perplexity
- Pretrained Encyclopedia: Weakly Supervised Knowledge-Pretrained Language Model
- Towards Interpretable Natural Language Understanding with Explanations as Latent Variables
- Factual Probing Is [MASK]: Learning vs. Learning to Recall
- Probing Neural Language Models for Human Tacit Assumptions
- Don't Say That! Making Inconsistent Dialogue Unlikely with Unlikelihood Training
- Probing and Fine-tuning Reading Comprehension Models for Few-shot Event Extraction
- Thinking Aloud: Dynamic Context Generation Improves Zero-Shot Reasoning Performance of GPT-2
- Pre-Trained Models: Past, Present and Future
- Behind the Scene: Revealing the Secrets of Pre-trained Vision-and-Language Models
- SBERT-WK: A Sentence Embedding Method by Dissecting BERT-based Word Models
- Scientific Language Models for Biomedical Knowledge Base Completion: An Empirical Study
- A Survey on Complex Knowledge Base Question Answering: Methods, Challenges and Solutions
- Combining pre-trained language models and structured knowledge
- Critical Thinking for Language Models
- GNN is a Counter? Revisiting GNN for Question Answering
- KgPLM: Knowledge-guided Language Model Pre-training via Generative and Discriminative Learning
- How to Query Language Models?
- Text Classification Using Label Names Only: A Language Model Self-Training Approach
- CBAG: Conditional Biomedical Abstract Generation
- Knowledge-driven Data Construction for Zero-shot Evaluation in Commonsense Question Answering
- Learning to Deceive Knowledge Graph Augmented Models via Targeted Perturbation
- Knowledge Injection into Dialogue Generation via Language Models
- Benchmarking neural embeddings for link prediction in knowledge graphs under semantic and structural changes
- Enriching a Model's Notion of Belief using a Persistent Memory
- Natural Language QA Approaches using Reasoning with External Knowledge
- An Empirical Study on Few-shot Knowledge Probing for Pretrained Language Models
- TransOMCS: From Linguistic Graphs to Commonsense Knowledge
- On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
- Unsupervised Pre-training for Natural Language Generation: A Literature Review
- Empower Entity Set Expansion via Language Model Probing
- Adaptive Semiparametric Language Models
- Language Model Evaluation in Open-ended Text Generation
- Recent Advances in Automated Question Answering In Biomedical Domain
- Probing Pre-Trained Language Models for Disease Knowledge
- SalKG: Learning From Knowledge Graph Explanations for Commonsense Reasoning
- Automatic Claim Review for Climate Science via Explanation Generation
- Self-supervised Text-to-SQL Learning with Header Alignment Training
- Zero-shot Slot Filling with DPR and RAG
- ForecastQA: A Question Answering Challenge for Event Forecasting with Temporal Text Data
- Navigating Human Language Models with Synthetic Agents
- Fast Model Editing at Scale
- Alleviating the Knowledge-Language Inconsistency: A Study for Deep Commonsense Knowledge
- SHARE: a System for Hierarchical Assistive Recipe Editing
- How BPE Affects Memorization in Transformers
- Few-shot Named Entity Recognition with Cloze Questions
- Modelling General Properties of Nouns by Selectively Averaging Contextualised Embeddings
- Evaluating German Transformer Language Models with Syntactic Agreement Tests
- CHIME: Cross-passage Hierarchical Memory Network for Generative Review Question Answering