Subtract, Transport, or Replay? Auditable Deletion from Language-Model Memory
arXiv:2607.27539
The paper studies how to precisely delete specific records from pretrained language models by either subtracting addressable memory entries or replaying model states, demonstrating that exact deletion is achievable when the memory representation permits it.
Abstract
Exact deletion from persistent language-model memory depends on whether a record's effect remains addressable after later computation. Native Kimi Delta Attention (KDA) gives a negative result for the tested receipt interface: the corpus-pooled raw recurrent contribution changes by 12-49% with the suffix and remains 8-49% after a decay-ledger correction. Native omission also changes later transition and write terms and other active caches. Frozen-input transport succeeds on its fixed-input control; the changed terms place native omission outside the tested receipt classes. Checkpoint replay supplies the evaluated recomputation path; zero residual on final logits and all 80 audited KDA arrays verifies restoration across the declared checkpoint surface. The complementary result is constructive. We retrofit support-vector memory into frozen Gemma 3 without attention transfer, low-rank recovery, distillation, adapters, or language-model parameter updates. Prefix-mass preservation and one box per prefix solve give base-matched admission at 4B with 1.85% perplexity overhead. At 1B and 4B, verified deletion agrees with its conditional retained-key refit within 1.3e-10 maximum next-token KL; behavioral attacks at 4B reach never-stored or chance baselines. Across 1B, 4B, and 12B, the 4B checkpoint uniquely combines base-matched admission with low overhead. The paper's two contributions are a negative result for native KDA's tested receipt classes and a positive training-free construction for addressable pretrained memory.
26 pages (9-page main text), 11 figures. v2: major revision. Replaces the trained-conversion pipeline with a training-free retrofit of the support-vector gate into frozen Gemma 3; adds the native Kimi Linear (KDA) receipt/forcing study and checkpoint-replay deletion audits on MIMIC. Earlier trained-conversion results are retained as legacy appendix diagnostics