94 citations · 301 across the 30 of their papers we have counts for
30 papers
Geometric Median Matching for Robust k-Subset Selection from Noisy Data
Anish Acharya, Sujay Sanghavi, Alexandros G. Dimakis +1
Data pruning -- the combinatorial task of selecting a small and representative subset from a large dataset, is crucial for mitigating the enormous computational costs associated wi…
Geometric Median (GM) Matching for Robust Data Pruning
Anish Acharya, Inderjit S Dhillon, Sujay Sanghavi
Large-scale data collections in the wild, are invariably noisy. Thus developing data pruning strategies that remain robust even in the presence of corruption is critical in practic…
Retraining with Predicted Hard Labels Provably Increases Model Accuracy
Rudrajit Das, Inderjit S. Dhillon, Alessandro Epasto +5
The performance of a model trained with noisy labels is often improved by simply \textit{retraining} the model with its \textit{own predicted hard labels} (i.e., 1/0 labels). Yet,…
When Attention Collapses: How Degenerate Layers in LLMs Enable Smaller, Stronger Models
Sunny Sanyal, Ravid Shwartz-Ziv, Alexandros G. Dimakis +1
Large Language Models (LLMs) are known for their performance, but we uncover a significant structural inefficiency: a phenomenon we term attention collapse. In many pre-trained dec…
Towards Quantifying the Preconditioning Effect of Adam
Rudrajit Das, Naman Agarwal, Sujay Sanghavi +1
There is a notable dearth of results characterizing the preconditioning effect of Adam and showing how it may alleviate the curse of ill-conditioning -- an issue plaguing gradient…
In-Context Learning with Transformers: Softmax Attention Adapts to Function Lipschitzness
Liam Collins, Advait Parulekar, Aryan Mokhtari +2
A striking property of transformers is their ability to perform in-context learning (ICL), a machine learning framework in which the learner is presented with a novel context durin…
Understanding the Training Speedup from Sampling with Approximate Losses
Rudrajit Das, Xi Chen, Bertram Ieong +2
It is well known that selecting samples with large losses/gradients can significantly reduce the number of training steps. However, the selection overhead is often too high to yiel…
Understanding Contrastive Representation Learning from Positive Unlabeled (PU) Data
Anish Acharya, Li Jing, Bhargav Bhushanam +4
Pretext Invariant Representation Learning (PIRL) followed by Supervised Fine-Tuning (SFT) has become a standard paradigm for learning with limited labels. We extend this approach t…
Pretrained deep models outperform GBDTs in Learning-To-Rank under label scarcity
Charlie Hou, Kiran Koshy Thekumparampil, Michael Shavlovsky +3
On tabular data, a significant body of literature has shown that current deep learning (DL) models perform at best similarly to Gradient Boosted Decision Trees (GBDTs), while signi…
Early Weight Averaging meets High Learning Rates for LLM Pre-training
Sunny Sanyal, Atula Neerkaje, Jean Kaddour +2
Training Large Language Models (LLMs) incurs significant cost; hence, any strategy that accelerates model convergence is helpful. In this paper, we investigate the ability of a sim…
Understanding Self-Distillation in the Presence of Label Noise
Rudrajit Das, Sujay Sanghavi
Self-distillation (SD) is the process of first training a \enquote{teacher} model and then using its predictions to train a \enquote{student} model with the \textit{same} architect…
Latent Variable Representation for Reinforcement Learning
Tongzheng Ren, Chenjun Xiao, Tianjun Zhang +5
Deep latent variable models have achieved significant empirical successes in model-based reinforcement learning (RL) due to their expressiveness in modeling complex transition dyna…
Toward Understanding Privileged Features Distillation in Learning-to-Rank
Shuo Yang, Sujay Sanghavi, Holakou Rahmanian +2
In learning-to-rank problems, a privileged feature is one that is available during model training, but not available at test time. Such features naturally arise in merchandised rec…
Positive Unlabeled Contrastive Learning
Anish Acharya, Sujay Sanghavi, Li Jing +4
Self-supervised pretraining on unlabeled data followed by supervised fine-tuning on labeled data is a popular paradigm for learning from limited labeled examples. We extend this pa…
Sample Efficiency of Data Augmentation Consistency Regularization
Shuo Yang, Yijun Dong, Rachel Ward +3
Data augmentation is popular in the training of large neural networks; currently, however, there is no clear theoretical comparison between different algorithmic choices on how to…
Combinatorial Bandits without Total Order for Arms
Shuo Yang, Tongzheng Ren, Inderjit S. Dhillon +1
We consider the combinatorial bandits problem, where at each time step, the online learner selects a size- subset from the arms set , where $\left|\mathcal{A}\r…
Linear Bandit Algorithms with Sublinear Time Complexity
Shuo Yang, Tongzheng Ren, Sanjay Shakkottai +3
We propose two linear bandits algorithms with per-step complexity sublinear in the number of arms . The algorithms are designed for applications where the arm set is extremely l…
Faster Non-Convex Federated Learning via Global and Local Momentum
Rudrajit Das, Anish Acharya, Abolfazl Hashemi +3
We propose \texttt{FedGLOMO}, a novel federated learning (FL) algorithm with an iteration complexity of to converge to an -stationary point (i.e., $\math…
On Generalization of Adaptive Methods for Over-parameterized Linear Regression
Vatsal Shah, Soumya Basu, Anastasios Kyrillidis +1
Over-parameterization and adaptive methods have played a crucial role in the success of deep learning in the last decade. The widespread use of over-parameterization has forced us…
On the Benefits of Multiple Gossip Steps in Communication-Constrained Decentralized Optimization
Abolfazl Hashemi, Anish Acharya, Rudrajit Das +3
In decentralized optimization, it is common algorithmic practice to have nodes interleave (local) gradient descent iterations with gossip (i.e. averaging over the network) steps. M…
Extreme Multi-label Classification from Aggregated Labels
Yanyao Shen, Hsiang-fu Yu, Sujay Sanghavi +1
Extreme multi-label classification (XMC) is the problem of finding the relevant labels for an input, from a very large universe of possible labels. We consider XMC in the setting w…
Choosing the Sample with Lowest Loss makes SGD Robust
Vatsal Shah, Xiaoxia Wu, Sujay Sanghavi
The presence of outliers can potentially significantly skew the parameters of machine learning models trained via stochastic gradient descent (SGD). In this paper we propose a simp…
Interaction Hard Thresholding: Consistent Sparse Quadratic Regression in Sub-quadratic Time and Space
Shuo Yang, Yanyao Shen, Sujay Sanghavi
Quadratic regression involves modeling the response as a (generalized) linear function of not only the features but also of quadratic terms . The inclusio…
Learning Distributions Generated by One-Layer ReLU Networks
Shanshan Wu, Alexandros G. Dimakis, Sujay Sanghavi
We consider the problem of estimating the parameters of a -dimensional rectified Gaussian distribution from i.i.d. samples. A rectified Gaussian distribution is defined by passi…