Showing 2024Show all
2 papers · 1 filter
cs.CL2024
Position Information Emerges in Causal Transformers Without Positional Encodings via Similarity of Nearby Embeddings
Chunsheng Zuo, Pavel Guerzhoy, Michael Guerzhoy
Transformers with causal attention can solve tasks that require positional information without using positional encodings. In this work, we propose and investigate a new hypothesis…
cs.LG2024
Breaking Symmetry When Training Transformers
Chunsheng Zuo, Michael Guerzhoy
As we show in this paper, the prediction for output token of Transformer architectures without one of the mechanisms of positional encodings and causal attention is invariant…