2 papers
cs.LG2025
To Infinity and Beyond: Tool-Use Unlocks Length Generalization in State Space Models
Eran Malach, Omid Saremi, Sinead Williamson +5
State Space Models (SSMs) have become the leading alternative to Transformers for sequence modeling. Their primary advantage is efficiency in long-context and long-form generation,…
cs.LG2024
How Far Can Transformers Reason? The Globality Barrier and Inductive Scratchpad
Emmanuel Abbe, Samy Bengio, Aryo Lotfi +2
Can Transformers predict new syllogisms by composing established ones? More generally, what type of targets can be learned by such models from scratch? Recent works show that Trans…