4 papers · 1 filter
A Memory Efficient Unified Algorithm for Online Learning of Linear Dynamical Systems
Yuval Ran-Milo, Angelos Assos, Elad Hazan
Motivated by the challenge of stabilizing a general unknown linear dynamical system (LDS) from observations, we study the natural prerequisite of online prediction. Our goal is to…
Attention Sinks Are Provably Necessary in Softmax Transformers: Evidence from Trigger-Conditional Tasks
Yuval Ran-Milo
Transformers often display an attention sink: probability mass concentrates on a fixed, content-agnostic position. Are sinks a byproduct of the optimization/training regime? Or are…
A Mechanistic Account of Attention Sinks in GPT-2: One Circuit, Broader Implications for Mitigation
Yuval Ran-Milo, Hila Ofek, Shahar Mendel
Transformers commonly exhibit an attention sink: disproportionately high attention to the first position. We study this behavior in GPT-2-style models with learned query biases and…
Outcome-Based RL Provably Leads Transformers to Reason, but Only With the Right Data
Yuval Ran-Milo, Yotam Alexander, Shahar Mendel +1
Transformers trained via Reinforcement Learning (RL) with outcome-based supervision can spontaneously develop the ability to generate intermediate reasoning steps (Chain-of-Thought…