Facts as Experts: Adaptable and Interpretable Neural Memory over Symbolic Knowledge

Abstract

Massive language models are the core of modern NLP modeling and have been shown to encode impressive amounts of commonsense and factual information. However, that knowledge exists only within the latent parameters of the model, inaccessible to inspection and interpretation, and even worse, factual information memorized from the training corpora is likely to become stale as the world changes. Knowledge stored as parameters will also inevitably exhibit all of the biases inherent in the source materials. To address these problems, we develop a neural language model that includes an explicit interface between symbolically interpretable factual information and subsymbolic neural knowledge. We show that this model dramatically improves performance on two knowledge-intensive question-answering tasks. More interestingly, the model can be updated without re-training by manipulating its symbolic representations. In particular this model allows us to add new facts and overwrite existing ones in ways that are not possible for earlier models.

References in corpus (11)

• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
• Language Models are Few-Shot Learners
• Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings
• REALM: Retrieval-Augmented Language Model Pre-Training
• Neural-Symbolic Learning and Reasoning: A Survey and Interpretation
• Tracking the World State with Recurrent Entity Networks
• Query2box: Reasoning over Knowledge Graphs in Vector Space using Box Embeddings
• Embedding Logical Queries on Knowledge Graphs
• Differentiable Reasoning over a Virtual Knowledge Base
• TensorLog: Deep Learning Meets Probabilistic DBs
• Faithful Embeddings for Knowledge Base Queries

Cited by in corpus (13)

• On the Opportunities and Risks of Foundation Models
• Modifying Memories in Transformer Models
• Towards Continual Knowledge Learning of Language Models
• Knowledge Enhanced Pretrained Language Models: A Compreshensive Survey
• Neural Language Generation: Formulation, Methods, and Evaluation
• Reasoning Over Virtual Knowledge Bases With Open Predicate Relations
• Iterative Hierarchical Attention for Answering Complex Questions over Long Documents
• Relational World Knowledge Representation in Contextual Language Models: A Review
• Highly Parallel Autoregressive Entity Linking with Discriminative Correction
• Relation-Guided Pre-Training for Open-Domain Question Answering
• CEAR: Cross-Entity Aware Reranker for Knowledge Base Completion
• Cross-Modal Retrieval Augmentation for Multi-Modal Classification
• Large Product Key Memory for Pretrained Language Models