4 papers
Structuring Sparsity: Block-Sparse Featurizers Capture Visual Concept Manifolds
Thomas Fel, Matthew Kowal, Mozes Jacobs +22
What is the geometry of a visual percept? The most widely used protocols for decomposing neural network representations into interpretable parts treat concepts as isolated directio…
A Unifying View of Attention Sinks: Two Algorithms, Two Solutions
Lukas Fesser, Mozes Jacobs, Thomas Fel +2
When attention concentrates on a single token, a sink, what is the model actually computing? Attention sinks are ubiquitous in softmax transformers, yet this shared visual signatur…
Block-Recurrent Dynamics in Vision Transformers
Mozes Jacobs, Thomas Fel, Richard Hakim +3
As Vision Transformers (ViTs) become standard vision backbones, a mechanistic account of their computational phenomenology is essential. Despite architectural cues that hint at dyn…
Traveling Waves Integrate Spatial Information Through Time
Mozes Jacobs, Roberto C. Budzinski, Lyle Muller +2
Traveling waves of neural activity are widely observed in the brain, but their precise computational function remains unclear. One prominent hypothesis is that they enable the tran…