Attention as In-Context Empirical Bayes: A Two-Stage View via Particle Dynamics

Matthew Smart, Soumya Ganguly, Nilava Metya +2

We study minimal attention-only transformers under all-token corruption and show they admit a two-stage empirical Bayes interpretation. A single attention step computes a kernel-we…