Cross-Attention and Encoder-Decoder Transformers: A Logical Characterization

Veeti Ahvonen, Damian Heiman, Antti Kuusisto +2

We give a novel logical characterization of encoder-decoder transformers, the foundational architecture for LLMs that also sees use in various settings that benefit from cross-atte…