20 citations · 23 across the 4 of their papers we have counts for
Showing 2023Show all
2 papers · 1 filter
cs.LG2023★ 2 cited
Self-Attention in Colors: Another Take on Encoding Graph Structure in Transformers
Romain Menegaux, Emmanuel Jehanno, Margot Selosse +1
We introduce a novel self-attention mechanism, which we call CSA (Chromatic Self-Attention), which extends the notion of attention scores to attention _filters_, independently modu…
cs.LG2023★ 1 cited
Rethinking Gauss-Newton for learning over-parameterized models
Michael Arbel, Romain Menegaux, Pierre Wolinski
This work studies the global convergence and implicit bias of Gauss Newton's (GN) when optimizing over-parameterized one-hidden layer networks in the mean-field regime. We first es…