Output Vector Editing for Memorization Mitigation in Large Language Models

Abstract

Large language models can reproduce training text, creating privacy, copyright, and security risks. We propose Output Vector Editing (OVE), a post-hoc method for suppressing text continuations that a model reproduces from a given prefix. Under superposition, a neuron can contribute to multiple features. Yet some neuron-level mitigation methods suppress localized neurons by zeroing their activations, removing each neuron's entire contribution. OVE instead edits their output vectors, combining (i) two-stage localization of MLP layers and neurons with (ii) full-vocabulary least-squares updates. Its multi-anchor variant, OVE-MA, adaptively selects continuation positions expected to provide stronger suppression relative to the required weight change. We evaluate recovery and locality across four models (360M--7B parameters), using autoregressively validated continuations and recovery tests with greedy decoding, beam search, and sampling. On 634 OLMo-7B pairs, OVE-MA with at most five output-vector updates per request leaves 24.45\% recoverable in these tests, compared with 32.02\% for the closest prior post-hoc editing method, REVS, while using approximately 75 times fewer update operations. These results show that adaptive position selection can concentrate effective suppression into a small number of weight updates. OVE edits weights without gradients and requires no additional intervention during inference.