6 papers
Exact-Solution Volume and Length Generalization in Transformers
Yijia Jessica Zhu, David Chiang
Research on transformer expressivity shows whether a transformer is capable of solving a given task, but gives little indication of whether the solution, if learned, is generalizab…
VC Dimension and Expressivity of Real-Valued Transformers
Gavin Dooley, Andy Yang, Yijia Jessica Zhu +3
Whereas previous results on abilities and limitations of transformers have restricted the definition of transformers in various ways, here we study softmax-attention, multi-layer t…
Length Generalization for Transformers via Compression
Georg Zetzsche, Hongjian Jiang, Andy Yang +4
Recent advancements in transformer length generalization theory enable us to reliably predict when a transformer can learn to solve a task. In particular, the C-RASP hypothesis (a…
Length Generalization Bounds for Transformers
Andy Yang, Pascal Bergsträßer, Georg Zetzsche +2
Length generalization is a key property of a learning algorithm that enables it to make correct predictions on inputs of any length, given finite training data. To provide such a g…
Probability Distributions Computed by Autoregressive Transformers
Andy Yang, Anej Svete, Jiaoda Li +4
Most expressivity results for transformers treat them as language recognizers -- devices that accept or reject strings -- rather than as they are used in practice: as language mode…
The Transformer Cookbook
Andy Yang, Christopher Watson, Anton Xue +6
We present the transformer cookbook: a collection of techniques for directly encoding algorithms into a transformer's parameters. This work addresses the steep learning curve of su…