4 papers
Unrolled Flow Models for Reasoning
Faissal Izermine, Hanru Bai, Oscar Davis +1
Flow matching enables language generation in few steps, but whether additional integration steps improve reasoning remains unclear. We prove that a flow parameterized by a two-laye…
Exact Symmetry as Algebra: Group-Structured Tensors that Enforce Physical Selection Rules
Paulina Hoyos, Shashanka Ubaru, Dongsung Huh +5
Symmetry determines which physical responses of molecules and crystals are allowed and underlies many models used throughout the physical sciences. Yet several widely used machine-…
Transformers Learn Faster with Semantic Focus
Parikshit Ram, Kenneth L. Clarkson, Tim Klinger +2
Various forms of sparse attention have been explored to mitigate the quadratic computational and memory cost of the attention mechanism in transformers. We study sparse transformer…
Transformer Circuits Can Realize Clustering Algorithms
Kenneth L. Clarkson, Lior Horesh, Takuya Ito +2
Although transformers are most commonly optimized as statistical sequence models, it is unclear to what extent they can implement and learn exact algorithmic computations. Here, we…