Projection-Free Transformers via Gaussian Kernel Attention

Debarshi Kundu, Archisman Ghosh, Swaroop Ghosh +1

Self-attention in Transformers is typically implemented as $\mathrm{softmax}(Q{K}^{\top }/\sqrt{d})V$, where $Q=X{W}_Q$, $K=X{W}_K$, and $V=X{W}_V$ are learned linear projections of the input…