Query Expansion and Key Specialization in Transformer Attention Geometry

Vidit Gupta, Siddhesh Nadkarni, Mihik Chaudhari +2

The projection of queries and keys are central to the attention mechanism in Transformer architectures. While they are mathematically symmetric, they play different roles in attent…