Robust Filter Attention: Self-Attention as Precision-Weighted State Estimation
arXiv:2509.04154
The paper presents Robust Filter Attention, a new view of self‑attention as a robust state estimator based on linear stochastic differential equations, and shows it improves language modeling perplexity and stability for longer contexts.
Abstract
We introduce Robust Filter Attention (RFA), a formulation of self-attention as a robust state estimator. Each token is treated as a noisy observation of a latent trajectory governed by a linear stochastic differential equation (SDE), and attention weights are determined by consistency under this model rather than static feature similarity. Under isotropic noise and decay assumptions, RFA matches the computational complexity of standard attention. On language modeling benchmarks, RFA achieves lower perplexity than RoPE within the training window while remaining stable under zero-shot extrapolation to longer contexts. The framework also provides a dynamical interpretation of standard positional mechanisms, connecting rotational embeddings and recency biases to transport and uncertainty propagation induced by stochastic dynamics.
Accepted to ICML 2026