22 citations · 22 across the 6 of their papers we have counts for
4 papers · 1 filter
Comparing Transformers and Hybrid Models at the Token Level
Yanhong Li, William Merrill
Hybrid language models that mix attention and recurrent layers have shown promise: theoretically, recurrent layers ameliorate the limitations of pure transformers on state tracking…
Efficiently Representing Algorithms With Chain-of-Thought Transformers
Yanhong Li, Anej Svete, Ashish Sabharwal +1
The increasing popularity of \emph{reasoning} models -- language models that output a series of reasoning or thought tokens before producing an answer -- is justified, in part, by…
Revisiting Padded Transformer Expressivity: Which Architectural Choices Matter and Which Don't
Anej Svete, William Merrill, Ryan Cotterell +1
Recent work describes what transformers can and cannot compute through connections to boolean circuits, but existing results lack exact characterizations and are sensitive to model…
Why Are Linear RNNs More Parallelizable?
William Merrill, Hongjian Jiang, Yanhong Li +2
The community is increasingly exploring linear RNNs (LRNNs) as language models, motivated by their expressive power and parallelizability. While prior work establishes the expressi…