Mass-Editing Memory in a Transformer

Abstract

Recent work has shown exciting promise in updating large language models with new memories, so as to replace obsolete information or add specialized knowledge. However, this line of work is predominantly limited to updating single associations. We develop MEMIT, a method for directly updating a language model with many memories, demonstrating experimentally that it can scale up to thousands of associations for GPT-J (6B) and GPT-NeoX (20B), exceeding prior work by orders of magnitude. Our code and data are at https://memit.baulab.info.

18 pages, 11 figures. Code and data at https://memit.baulab.info

Cited by in corpus (4)

• CommonsenseVIS: Visualizing and Understanding Commonsense Reasoning Capabilities of Natural Language Models
• Future Lens: Anticipating Subsequent Tokens from a Single Hidden State
• Language Anisotropic Cross-Lingual Model Editing
• A Glitch in the Matrix? Locating and Detecting Language Model Grounding with Fakepedia