4 citations · 8 across the 11 of their papers we have counts for
3 papers · 1 filter
The Effect of Training Task Diversity on In-Context Learning through the Lens of Low-Dimensional Subspaces
Soo Min Kwon, Alec S. Xu, Can Yaras +3
The transformer's emergent ability to perform in-context learning (ICL) has sparked a wide range of studies designed to understand its underlying mechanisms. Existing works often s…
Stochastic Sparse Attention for Memory-Bound Inference
Kyle Lee, Corentin Delacour, Kevin Callahan-Coray +5
Autoregressive decoding becomes bandwidth-limited at long contexts, as generating each token requires reading all key and value vectors from KV cache. We present Stochastic A…
Emergent Low-Rank Training Dynamics in MLPs with Smooth Activations
Alec S. Xu, Can Yaras, Matthew Asato +2
Recent empirical evidence has demonstrated that the training dynamics of large-scale deep neural networks occur within low-dimensional subspaces. While this has inspired new resear…