2 citations · 3 across the 13 of their papers we have counts for
17 papers
Transformers Learn Two Competing Sorting Algorithms, and Only One Length-Generalizes
Khashayar Gatmiry, Nathan W. Henry, Chloe H. Su +5
We show that two-layer transformers trained to near-zero loss on a sorting task converge to one of two mechanistically distinct solution classes depending on their initialization.…
Evolving Towards Better Codes: LLM-Guided Search for High-Distance Binary Linear Codes
Amal Seddas, Vladyslav Shashkov, Maryna Viazovska +1
Evolutionary program search driven by large language models (LLMs) has produced record-breaking constructions for open problems in combinatorics and beyond. We apply this approach…
Learn Your Own Thoughts: Abstract Token Curriculum
Khashayar Gatmiry, Avrajit Ghosh, Parsa Mirtaheri +4
Large Language Models (LLMs) have achieved remarkable reasoning capabilities by utilizing chain-of-thought (CoT) as a scratchpad for intermediate stages of thinking. However, CoT t…
On the Importance of Gating: Memorization vs. In-Context Learning in State Space Models
William L. Tong, Aryo Lotfi, Emmanuel Abbe +6
State Space Models (SSMs) have emerged as a compelling alternative to Transformers, enabling sequence modeling with constant memory and linear compute. Although SSMs exhibit reason…
Soft Guidance Starts to Outperform CoT Prompting as LLMs Improve
Denys Pushkin, Albert Q. Jiang, Aryo Lotfi +2
Chain-of-Thought (CoT) prompting remains the standard baseline for evaluating models' reasoning abilities. Originally, this technique was introduced to elicit step-by-step reasonin…
Blind-Spots-Bench: Evaluating Blind Spots in Multimodal Models
Matteo Santelmo, Xiuying Wei, Israa Fakih +5
Modern AI models achieve strong performance on many established benchmarks, yet they still fail on tasks that humans find almost trivial, such as manipulating a string or drawing a…
LEAD: Breaking the No-Recovery Bottleneck in Long-Horizon Reasoning
Denys Pushkin, Emmanuel Abbe
Long-horizon execution in Large Language Models (LLMs) remains unstable even when high-level strategies are provided. Evaluating on controlled algorithmic puzzles, we demonstrate t…
Goldilocks RL: Tuning Task Difficulty to Escape Sparse Rewards for Reasoning
Ilia Mahrooghi, Aryo Lotfi, Emmanuel Abbe
Reinforcement learning has emerged as a powerful paradigm for unlocking reasoning capabilities in language models. However, relying on sparse rewards makes this process highly samp…
To Infinity and Beyond: Tool-Use Unlocks Length Generalization in State Space Models
Eran Malach, Omid Saremi, Sinead Williamson +5
State Space Models (SSMs) have become the leading alternative to Transformers for sequence modeling. Their primary advantage is efficiency in long-context and long-form generation,…
RL for Reasoning by Adaptively Revealing Rationales
Mohammad Hossein Amani, Aryo Lotfi, Nicolas Mario Baldwin +4
Learning in the combinatorially large output space of sequence generation problems is challenging as providing expert demonstrations scales poorly with sequence length, and RL stru…
AbstRaL: Augmenting LLMs' Reasoning by Reinforcing Abstract Thinking
Silin Gao, Antoine Bosselut, Samy Bengio +1
Recent studies have shown that large language models (LLMs), especially smaller ones, often lack robustness in grade school math (GSM) reasoning. In particular, they tend to experi…
Algorithm Discovery With LLMs: Evolutionary Search Meets Reinforcement Learning
Anja Surina, Amin Mansouri, Lars Quaedvlieg +4
Discovering efficient algorithms for solving complex problems has been an outstanding challenge in mathematics and computer science, requiring substantial human expertise over the…
What Makes the Preferred Thinking Direction for LLMs in Multiple-choice Questions?
Yizhe Zhang, Richard Bai, Zijin Gu +5
Language models usually use left-to-right (L2R) autoregressive factorization. However, L2R factorization may not always be the best inductive bias. Therefore, we investigate whethe…
Chain-of-Sketch: Enabling Global Visual Reasoning
Aryo Lotfi, Enrico Fini, Samy Bengio +2
Modern vision models have achieved remarkable success in benchmarks where local features provide critical information about the target. There is now a growing interest in tackling…
On the Minimal Degree Bias in Generalization on the Unseen for non-Boolean Functions
Denys Pushkin, Raphaël Berthier, Emmanuel Abbe
We investigate the out-of-domain generalization of random feature (RF) models and Transformers. We first prove that in the `generalization on the unseen (GOTU)' setting, where trai…
How Far Can Transformers Reason? The Globality Barrier and Inductive Scratchpad
Emmanuel Abbe, Samy Bengio, Aryo Lotfi +2
Can Transformers predict new syllogisms by composing established ones? More generally, what type of targets can be learned by such models from scratch? Recent works show that Trans…
When can transformers reason with abstract symbols?
Enric Boix-Adsera, Omid Saremi, Emmanuel Abbe +3
We investigate the capabilities of transformer models on relational reasoning tasks. In these tasks, models are trained on a set of strings encoding abstract relations, and are the…