Encoder-Decoder Transformers: Logical Characterizations and Periodicity

Veeti Ahvonen, Damian Heiman, Antti Kuusisto +2

We give logical characterizations of encoder-decoder transformers, the foundational architecture for LLMs that also sees use in various settings that benefit from cross-attention,…