A Probabilistic Interpretation of Transformers

Alexander Shim

We propose a probabilistic interpretation of exponential dot product attention of transformers and contrastive learning based off of exponential families. The attention sublayer of…