2 citations · 2 across the 3 of their papers we have counts for
Showing 2024Show all
2 papers · 1 filter
cs.LG2024★ 2 cited
Dynamic metastability in the self-attention model
Borjan Geshkovski, Hugo Koubbi, Yury Polyanskiy +1
We consider the self-attention model - an interacting particle system on the unit sphere, which serves as a toy model for Transformers, the deep neural network architecture behind…
cs.LG2024
Understanding Catastrophic Forgetting In LoRA via Mean-Field Attention Dynamics
Hugo Koubbi, Louis Hernandez, Matthieu Boussard
Low-Rank Adaptation (LoRA) is the dominant parameter-efficient fine-tuning method due to its favorable compute-performance trade-off, yet it suffers from catastrophic forgetting. W…