On attention heads and bilinear forms

Andrew O'Desky

We study the symmetric and antisymmetric parts of bilinear forms in the attention heads of trained large language models. We introduce an orthogonally invariant profile map from re…