4 citations · 8 across the 11 of their papers we have counts for
12 papers
The Effect of Training Task Diversity on In-Context Learning through the Lens of Low-Dimensional Subspaces
Soo Min Kwon, Alec S. Xu, Can Yaras +3
The transformer's emergent ability to perform in-context learning (ICL) has sparked a wide range of studies designed to understand its underlying mechanisms. Existing works often s…
Stochastic Sparse Attention for Memory-Bound Inference
Kyle Lee, Corentin Delacour, Kevin Callahan-Coray +5
Autoregressive decoding becomes bandwidth-limited at long contexts, as generating each token requires reading all key and value vectors from KV cache. We present Stochastic A…
Emergent Low-Rank Training Dynamics in MLPs with Smooth Activations
Alec S. Xu, Can Yaras, Matthew Asato +2
Recent empirical evidence has demonstrated that the training dynamics of large-scale deep neural networks occur within low-dimensional subspaces. While this has inspired new resear…
MonarchAttention: Zero-Shot Conversion to Fast, Hardware-Aware Structured Attention
Can Yaras, Alec S. Xu, Pierre Abillama +2
Transformers have achieved state-of-the-art performance across various tasks, but suffer from a notable quadratic complexity in sequence length due to the attention mechanism. In t…
Out-of-Distribution Generalization of In-Context Learning: A Low-Dimensional Subspace Perspective
Soo Min Kwon, Alec S. Xu, Can Yaras +2
The transformer's remarkable ability to perform in-context learning (ICL) has sparked a wide range of studies designed to understand its strengths and limitations. However, a theor…
An Overview of Low-Rank Structures in the Training and Adaptation of Large Models
Laura Balzano, Tianjiao Ding, Benjamin D. Haeffele +5
The substantial computational demands of modern large-scale deep learning present significant challenges for efficient training and deployment. Recent research has revealed a wides…
Linearly Separable Features in Shallow Nonlinear Networks: Width Scales Polynomially with Intrinsic Data Dimension
Alec S. Xu, Can Yaras, Peng Wang +1
Deep neural networks have attained remarkable success across diverse classification tasks. Recent empirical studies have shown that deep networks learn features that are linearly s…
Explaining and Mitigating the Modality Gap in Contrastive Multimodal Learning
Can Yaras, Siyi Chen, Peng Wang +1
Multimodal learning has recently gained significant popularity, demonstrating impressive performance across various zero-shot classification tasks and a range of perceptive and gen…
Compressible Dynamics in Deep Overparameterized Low-Rank Learning & Adaptation
Can Yaras, Peng Wang, Laura Balzano +1
While overparameterization in machine learning models offers great benefits in terms of optimization and generalization, it also leads to increased computational requirements as mo…
Understanding Deep Representation Learning via Layerwise Feature Compression and Discrimination
Peng Wang, Xiao Li, Can Yaras +4
Over the past decade, deep learning has proven to be a highly effective tool for learning meaningful features from raw data. However, it remains an open question how deep networks…
The Law of Parsimony in Gradient Descent for Learning Deep Linear Networks
Can Yaras, Peng Wang, Wei Hu +3
Over the past few years, an extensively studied phenomenon in training deep networks is the implicit bias of gradient descent towards parsimonious solutions. In this work, we inves…
Neural Collapse with Normalized Features: A Geometric Analysis over the Riemannian Manifold
Can Yaras, Peng Wang, Zhihui Zhu +2
When training overparameterized deep networks for classification tasks, it has been widely observed that the learned features exhibit a so-called "neural collapse" phenomenon. More…