4 papers
Dynamical Kernel Attention: Estimating the Present from a Record of the Past
Peter Racioppo
Standard self-attention compares tokens by similarity but does not account for how much of a past state survives to the present. We introduce Dynamical Kernel Attention (DKA), whic…
The Transformer as a Polar State Estimator
Peter Racioppo
We show that the core components of the Transformer---attention, residual connections, and normalization---arise naturally from a single geometric state estimation problem. Modelin…
AC-SINDy: Compositional Sparse Identification of Nonlinear Dynamics
Peter Racioppo
We present AC-SINDy, a compositional extension of the Sparse Identification of Nonlinear Dynamics (SINDy) framework that replaces explicit feature libraries with a structured repre…
Robust Filter Attention: Self-Attention as Precision-Weighted State Estimation
Peter Racioppo
We introduce Robust Filter Attention (RFA), a formulation of self-attention as a robust state estimator. Each token is treated as a noisy observation of a latent trajectory governe…