2 citations · 3 across the 13 of their papers we have counts for
9 papers · 1 filter
Transformers Learn Two Competing Sorting Algorithms, and Only One Length-Generalizes
Khashayar Gatmiry, Nathan W. Henry, Chloe H. Su +5
We show that two-layer transformers trained to near-zero loss on a sorting task converge to one of two mechanistically distinct solution classes depending on their initialization.…
Learn Your Own Thoughts: Abstract Token Curriculum
Khashayar Gatmiry, Avrajit Ghosh, Parsa Mirtaheri +4
Large Language Models (LLMs) have achieved remarkable reasoning capabilities by utilizing chain-of-thought (CoT) as a scratchpad for intermediate stages of thinking. However, CoT t…
On the Importance of Gating: Memorization vs. In-Context Learning in State Space Models
William L. Tong, Aryo Lotfi, Emmanuel Abbe +6
State Space Models (SSMs) have emerged as a compelling alternative to Transformers, enabling sequence modeling with constant memory and linear compute. Although SSMs exhibit reason…
Goldilocks RL: Tuning Task Difficulty to Escape Sparse Rewards for Reasoning
Ilia Mahrooghi, Aryo Lotfi, Emmanuel Abbe
Reinforcement learning has emerged as a powerful paradigm for unlocking reasoning capabilities in language models. However, relying on sparse rewards makes this process highly samp…
To Infinity and Beyond: Tool-Use Unlocks Length Generalization in State Space Models
Eran Malach, Omid Saremi, Sinead Williamson +5
State Space Models (SSMs) have become the leading alternative to Transformers for sequence modeling. Their primary advantage is efficiency in long-context and long-form generation,…
RL for Reasoning by Adaptively Revealing Rationales
Mohammad Hossein Amani, Aryo Lotfi, Nicolas Mario Baldwin +4
Learning in the combinatorially large output space of sequence generation problems is challenging as providing expert demonstrations scales poorly with sequence length, and RL stru…
Chain-of-Sketch: Enabling Global Visual Reasoning
Aryo Lotfi, Enrico Fini, Samy Bengio +2
Modern vision models have achieved remarkable success in benchmarks where local features provide critical information about the target. There is now a growing interest in tackling…
On the Minimal Degree Bias in Generalization on the Unseen for non-Boolean Functions
Denys Pushkin, Raphaël Berthier, Emmanuel Abbe
We investigate the out-of-domain generalization of random feature (RF) models and Transformers. We first prove that in the `generalization on the unseen (GOTU)' setting, where trai…
How Far Can Transformers Reason? The Globality Barrier and Inductive Scratchpad
Emmanuel Abbe, Samy Bengio, Aryo Lotfi +2
Can Transformers predict new syllogisms by composing established ones? More generally, what type of targets can be learned by such models from scratch? Recent works show that Trans…