1 citations · 1 across the 8 of their papers we have counts for
4 papers · 1 filter
VC Dimension and Expressivity of Real-Valued Transformers
Gavin Dooley, Andy Yang, Yijia Jessica Zhu +3
Whereas previous results on abilities and limitations of transformers have restricted the definition of transformers in various ways, here we study softmax-attention, multi-layer t…
Length Generalization for Transformers via Compression
Georg Zetzsche, Hongjian Jiang, Andy Yang +4
Recent advancements in transformer length generalization theory enable us to reliably predict when a transformer can learn to solve a task. In particular, the C-RASP hypothesis (a…
Algebraic Decomposition Theory for Transformer Length Generalization
Andy Yang, Blerta Veseli, Corentin Barloy +5
Transformer-based language models are known to sometimes generalize to sequences longer than seen during training, but we lack a precise characterization of which tasks admit lengt…
Length Generalization Bounds for Transformers
Andy Yang, Pascal Bergsträßer, Georg Zetzsche +2
Length generalization is a key property of a learning algorithm that enables it to make correct predictions on inputs of any length, given finite training data. To provide such a g…