94 citations · 301 across the 30 of their papers we have counts for
7 papers · 1 filter
Geometric Median (GM) Matching for Robust Data Pruning
Anish Acharya, Inderjit S Dhillon, Sujay Sanghavi
Large-scale data collections in the wild, are invariably noisy. Thus developing data pruning strategies that remain robust even in the presence of corruption is critical in practic…
Retraining with Predicted Hard Labels Provably Increases Model Accuracy
Rudrajit Das, Inderjit S. Dhillon, Alessandro Epasto +5
The performance of a model trained with noisy labels is often improved by simply \textit{retraining} the model with its \textit{own predicted hard labels} (i.e., 1/0 labels). Yet,…
When Attention Collapses: How Degenerate Layers in LLMs Enable Smaller, Stronger Models
Sunny Sanyal, Ravid Shwartz-Ziv, Alexandros G. Dimakis +1
Large Language Models (LLMs) are known for their performance, but we uncover a significant structural inefficiency: a phenomenon we term attention collapse. In many pre-trained dec…
Towards Quantifying the Preconditioning Effect of Adam
Rudrajit Das, Naman Agarwal, Sujay Sanghavi +1
There is a notable dearth of results characterizing the preconditioning effect of Adam and showing how it may alleviate the curse of ill-conditioning -- an issue plaguing gradient…
In-Context Learning with Transformers: Softmax Attention Adapts to Function Lipschitzness
Liam Collins, Advait Parulekar, Aryan Mokhtari +2
A striking property of transformers is their ability to perform in-context learning (ICL), a machine learning framework in which the learner is presented with a novel context durin…
Understanding the Training Speedup from Sampling with Approximate Losses
Rudrajit Das, Xi Chen, Bertram Ieong +2
It is well known that selecting samples with large losses/gradients can significantly reduce the number of training steps. However, the selection overhead is often too high to yiel…
Understanding Contrastive Representation Learning from Positive Unlabeled (PU) Data
Anish Acharya, Li Jing, Bhargav Bhushanam +4
Pretext Invariant Representation Learning (PIRL) followed by Supervised Fine-Tuning (SFT) has become a standard paradigm for learning with limited labels. We extend this approach t…