Spectral Geometry of Attention: From Information Routing to Uncertainty

Giulio Viganò, Simone Melzi, Maks Ovsjanikov

In this work, we study transformer attention through the lens of spectral geometry and operator theory. We view each attention head as a functional map between Hilbert spaces of fu…