3 citations · 6 across the 6 of their papers we have counts for
4 papers · 1 filter
Weight decay induces low-rank attention layers
Seijin Kobayashi, Yassir Akram, Johannes Von Oswald
The effect of regularizers such as weight decay when training deep neural networks is not well understood. We study the influence of weight decay as well as -regularization whe…
Learning Randomized Algorithms with Transformers
Johannes von Oswald, Seijin Kobayashi, Yassir Akram +1
Randomization is a powerful tool that endows algorithms with remarkable properties. For instance, randomized algorithms excel in adversarial settings, often surpassing the worst-ca…
When can transformers compositionally generalize in-context?
Seijin Kobayashi, Simon Schug, Yassir Akram +5
Many tasks can be composed from a few independent components. This gives rise to a combinatorial explosion of possible tasks, only some of which might be encountered during trainin…
Attention as a Hypernetwork
Simon Schug, Seijin Kobayashi, Yassir Akram +2
Transformers can under some circumstances generalize to novel problem instances whose constituent parts might have been encountered during training, but whose compositions have not…