5 papers
Dynamical principles of habituation across substrates and scales
Matthew Smart, Stanislav Y. Shvartsman, Martin Mönnigmann
Habituation is a basic form of learning in which a system's response to repeated stimulation progressively diminishes but eventually recovers when the stimulus is withheld. Long st…
Attention as In-Context Empirical Bayes: A Two-Stage View via Particle Dynamics
Matthew Smart, Soumya Ganguly, Nilava Metya +2
We study minimal attention-only transformers under all-token corruption and show they admit a two-stage empirical Bayes interpretation. A single attention step computes a kernel-we…
In-context denoising with one-layer transformers: connections between attention and associative memory retrieval
Matthew Smart, Alberto Bietti, Anirvan M. Sengupta
We introduce in-context denoising, a task that refines the connection between attention-based architectures and dense associative memory (DAM) networks, also known as modern Hopfie…
Minimal motifs for habituating systems
Matthew Smart, Stanislav Y. Shvartsman, Martin Mönnigmann
Habituation - a phenomenon in which a dynamical system exhibits a diminishing response to repeated stimulations that eventually recovers when the stimulus is withheld - is universa…
A minimal dynamical system and analog circuit for non-associative learning
Matthew Smart, Stanislav Y. Shvartsman, Martin Mönnigmann
Learning in living organisms is typically associated with networks of neurons. The use of large numbers of adjustable units has also been a crucial factor in the continued success…