3 papers
cs.LG2026
A Unifying View of Attention Sinks: From Mechanisms to Architectural Interventions
Lukas Fesser, Lukas Fesser*, Mozes Jacobs* +5
When attention concentrates on a single token, a sink, what is the model actually computing? Attention sinks are ubiquitous in softmax transformers, yet this shared visual signatur…
cs.CV2025
Block-Recurrent Dynamics in Vision Transformers
Mozes Jacobs, Thomas Fel, Richard Hakim +3
As Vision Transformers (ViTs) become standard vision backbones, a mechanistic account of their computational phenomenology is essential. Despite architectural cues that hint at dyn…
cs.NE2025
Bridging Expressivity and Scalability with Adaptive Unitary SSMs
Arjun Karuvally, Franz Nowak, Anderson T. Keller +3
Recent work has revealed that state space models (SSMs), while efficient for long-sequence processing, are fundamentally limited in their ability to represent formal languages-part…