2 citations · 7 across the 18 of their papers we have counts for
11 papers · 1 filter
Efficiently Representing Algorithms With Chain-of-Thought Transformers
Yanhong Li, Anej Svete, Ashish Sabharwal +1
The increasing popularity of \emph{reasoning} models -- language models that output a series of reasoning or thought tokens before producing an answer -- is justified, in part, by…
Revisiting Padded Transformer Expressivity: Which Architectural Choices Matter and Which Don't
Anej Svete, William Merrill, Ryan Cotterell +1
Recent work describes what transformers can and cannot compute through connections to boolean circuits, but existing results lack exact characterizations and are sensitive to model…
Olmo Hybrid: From Theory to Practice and Back
William Merrill, Yanhong Li, Tyler Romero +19
Recent work has demonstrated the potential of non-transformer language models, especially linear recurrent neural networks (RNNs) and hybrid models that mix recurrence and attentio…
Why Are Linear RNNs More Parallelizable?
William Merrill, Hongjian Jiang, Yanhong Li +2
The community is increasingly exploring linear RNNs (LRNNs) as language models, motivated by their expressive power and parallelizability. While prior work establishes the expressi…
On the Reasoning Abilities of Masked Diffusion Language Models
Anej Svete, Ashish Sabharwal
Masked diffusion models (MDMs) for text offer a compelling alternative to traditional autoregressive language models. Parallel generation makes them efficient, but their computatio…
Motivating Next-Gen Accelerators with Flexible (N:M) Activation Sparsity via Benchmarking Lightweight Post-Training Sparsification Approaches
Shirin Alanova, Kristina Kazistova, Ekaterina Galaeva +7
The demand for efficient large language model (LLM) inference has intensified the focus on sparsification techniques. While semi-structured (N:M) pruning is well-established for we…
AutoDiscovery: Open-ended Scientific Discovery via Bayesian Surprise
Dhruv Agarwal, Bodhisattwa Prasad Majumder, Reece Adamson +8
The promise of autonomous scientific discovery (ASD) hinges not only on answering questions, but also on knowing which questions to ask. Most recent works in ASD explore the use of…
Exact Expressive Power of Transformers with Padding
William Merrill, Ashish Sabharwal
Chain of thought is a natural inference-time method for increasing the computational power of transformer-based large language models (LLMs), but comes at the cost of sequential de…
A Little Depth Goes a Long Way: The Expressive Power of Log-Depth Transformers
William Merrill, Ashish Sabharwal
Recent theoretical results show transformers cannot express sequential reasoning problems over long inputs, intuitively because their computational depth is bounded. However, prior…
The Illusion of State in State-Space Models
William Merrill, Jackson Petty, Ashish Sabharwal
State-space models (SSMs) have emerged as a potential alternative architecture for building large language models (LLMs) compared to the previously ubiquitous transformer architect…
The Expressive Power of Transformers with Chain of Thought
William Merrill, Ashish Sabharwal
Recent theoretical work has identified surprisingly simple reasoning problems, such as checking if two nodes in a graph are connected or simulating finite-state machines, that are…