1 citations · 4 across the 27 of their papers we have counts for
5 papers · 1 filter
Probability Distributions Computed by Autoregressive Transformers
Andy Yang, Anej Svete, Jiaoda Li +4
Most expressivity results for transformers treat them as language recognizers -- devices that accept or reject strings -- rather than as they are used in practice: as language mode…
On the Reasoning Abilities of Masked Diffusion Language Models
Anej Svete, Ashish Sabharwal
Masked diffusion models (MDMs) for text offer a compelling alternative to traditional autoregressive language models. Parallel generation makes them efficient, but their computatio…
The Transformer Cookbook
Andy Yang, Christopher Watson, Anton Xue +6
We present the transformer cookbook: a collection of techniques for directly encoding algorithms into a transformer's parameters. This work addresses the steep learning curve of su…
Information Locality as an Inductive Bias for Neural Language Models
Taiga Someya, Anej Svete, Brian DuSell +3
Inductive biases are inherent in every machine learning system, shaping how models generalize from finite data. In the case of neural language models (LMs), debates persist as to w…
Unique Hard Attention: A Tale of Two Sides
Selim Jerad, Anej Svete, Jiaoda Li +1
Understanding the expressive power of transformers has recently attracted attention, as it offers insights into their abilities and limitations. Many studies analyze unique hard at…