Algebraic Invariants of Lightning Self-Attention

Yulia Alexandr, Hao Duan, Guido Montúfar

We study the polynomial coefficients of lightning self-attention as coordinates of an algebraic variety. In the single-token case, we characterize the coefficient variety as a rank…