In-context denoising with one-layer transformers: connections between attention and associative memory retrieval

Matthew Smart, Alberto Bietti, Anirvan M. Sengupta

We introduce in-context denoising, a task that refines the connection between attention-based architectures and dense associative memory (DAM) networks, also known as modern Hopfie…