Probabilistic Transformers

Abstract

We show that Transformers are Maximum Posterior Probability estimators for Mixtures of Gaussian Models. This brings a probabilistic point of view to Transformers and suggests extensions to other probabilistic cases.

References in corpus (4)

• Neural Machine Translation by Jointly Learning to Align and Translate
• Hopfield Networks is All You Need
• The Lipschitz Constant of Self-Attention
• Attention that does not Explain Away