Locating and Editing Factual Associations in GPT

Abstract

We analyze the storage and recall of factual associations in autoregressive transformer language models, finding evidence that these associations correspond to localized, directly-editable computations. We first develop a causal intervention for identifying neuron activations that are decisive in a model's factual predictions. This reveals a distinct set of steps in middle-layer feed-forward modules that mediate factual predictions while processing subject tokens. To test our hypothesis that these computations correspond to factual association recall, we modify feed-forward weights to update specific factual associations using Rank-One Model Editing (ROME). We find that ROME is effective on a standard zero-shot relation extraction (zsRE) model-editing task, comparable to existing methods. To perform a more sensitive evaluation, we also evaluate ROME on a new dataset of counterfactual assertions, on which it simultaneously maintains both specificity and generalization, whereas other methods sacrifice one or another. Our results confirm an important role for mid-layer feed-forward modules in storing factual associations and suggest that direct manipulation of computational mechanisms may be a feasible approach for model editing. The code, dataset, visualizations, and an interactive demo notebook are available at https://rome.baulab.info/

NeurIPS 2022. 35 pages, 30 figures. Code and data at https://rome.baulab.info/

Cited by in corpus (19)

• A Comprehensive Survey of Large Language Models and Multimodal Large Language Models in Medicine
• A Survey on Symbolic Knowledge Distillation of Large Language Models
• WikiChat: Stopping the Hallucination of Large Language Model Chatbots by Few-Shot Grounding on Wikipedia
• Grounding and Evaluation for Large Language Models: Practical Challenges and Lessons Learned (Survey)
• Inseq: An Interpretability Toolkit for Sequence Generation Models
• Prompt Perturbation in Retrieval-Augmented Generation based Large Language Models
• On Early Detection of Hallucinations in Factual Question Answering
• The Geometry of Concepts: Sparse Autoencoder Feature Structure
• Cross-Lingual Consistency of Factual Knowledge in Multilingual Language Models
• Balancing User Preferences by Social Networks: A Condition-Guided Social Recommendation Model for Mitigating Popularity Bias
• Exploring the Reversal Curse and Other Deductive Logical Reasoning in BERT and GPT-Based Large Language Models
• Modularity in Deep Learning: A Survey
• Future Lens: Anticipating Subsequent Tokens from a Single Hidden State
• Adaptive Activation Steering: A Tuning-Free LLM Truthfulness Improvement Method for Diverse Hallucinations Categories
• Fairness Mediator: Neutralize Stereotype Associations to Mitigate Bias in Large Language Models
• Which Neurons Matter in IR? Applying Integrated Gradients-based Methods to Understand Cross-Encoders
• Explaining Relation Classification Models with Semantic Extents
• A Glitch in the Matrix? Locating and Detecting Language Model Grounding with Fakepedia
• Vectorizing string entries for data processing on tables: when are larger language models better?