22 citations · 22 across the 6 of their papers we have counts for
5 papers · 1 filter
Efficiently Representing Algorithms With Chain-of-Thought Transformers
Yanhong Li, Anej Svete, Ashish Sabharwal +1
The increasing popularity of \emph{reasoning} models -- language models that output a series of reasoning or thought tokens before producing an answer -- is justified, in part, by…
Revisiting Padded Transformer Expressivity: Which Architectural Choices Matter and Which Don't
Anej Svete, William Merrill, Ryan Cotterell +1
Recent work describes what transformers can and cannot compute through connections to boolean circuits, but existing results lack exact characterizations and are sensitive to model…
Why Are Linear RNNs More Parallelizable?
William Merrill, Hongjian Jiang, Yanhong Li +2
The community is increasingly exploring linear RNNs (LRNNs) as language models, motivated by their expressive power and parallelizability. While prior work establishes the expressi…
The Transformer Cookbook
Andy Yang, Christopher Watson, Anton Xue +6
We present the transformer cookbook: a collection of techniques for directly encoding algorithms into a transformer's parameters. This work addresses the steep learning curve of su…
What Formal Languages Can Transformers Express? A Survey
Lena Strobl, William Merrill, Gail Weiss +2
As transformers have gained prominence in natural language processing, some researchers have investigated theoretically what problems they can and cannot solve, by treating problem…