1 citations · 1 across the 6 of their papers we have counts for
5 papers · 1 filter
Unrolled Flow Models for Reasoning
Faissal Izermine, Hanru Bai, Oscar Davis +1
Flow matching enables language generation in few steps, but whether additional integration steps improve reasoning remains unclear. We prove that a flow parameterized by a two-laye…
Universal Transformers for Circuit Computations: Perfect Length Generalization in Tiny Transformers
Takuya Ito, Ruchir Puri, Murray Campbell +1
Learning generalizable algorithmic computations remains a challenge for neural networks, as reflected in persistent failures on compositional and length generalization benchmarks.…
Transformer Circuits Can Realize Clustering Algorithms
Kenneth L. Clarkson, Lior Horesh, Takuya Ito +2
Although transformers are most commonly optimized as statistical sequence models, it is unclear to what extent they can implement and learn exact algorithmic computations. Here, we…
Learning interpretable positional encodings in transformers depends on initialization
Takuya Ito, Luca Cocchi, Tim Klinger +3
In transformers, the positional encoding (PE) provides essential information that distinguishes the position and order amongst tokens in a sequence. Most prior investigations of PE…
On the generalization capacity of neural networks during generic multimodal reasoning
Takuya Ito, Soham Dan, Mattia Rigotti +2
The advent of the Transformer has led to the development of large language models (LLM), which appear to demonstrate human-like capabilities. To assess the generality of this class…