Uncovering hidden geometry in Transformers via disentangling position and context

Jiajun Song, Yiqiao Zhong

Transformers are widely used to extract semantic meanings from input tokens, yet they usually operate as black-box models. In this paper, we present a simple yet informative decomp…