94 citations · 301 across the 30 of their papers we have counts for
23 papers · 1 filter
Geometric Median Matching for Robust k-Subset Selection from Noisy Data
Anish Acharya, Sujay Sanghavi, Alexandros G. Dimakis +1
Data pruning -- the combinatorial task of selecting a small and representative subset from a large dataset, is crucial for mitigating the enormous computational costs associated wi…
Geometric Median (GM) Matching for Robust Data Pruning
Anish Acharya, Inderjit S Dhillon, Sujay Sanghavi
Large-scale data collections in the wild, are invariably noisy. Thus developing data pruning strategies that remain robust even in the presence of corruption is critical in practic…
Retraining with Predicted Hard Labels Provably Increases Model Accuracy
Rudrajit Das, Inderjit S. Dhillon, Alessandro Epasto +5
The performance of a model trained with noisy labels is often improved by simply \textit{retraining} the model with its \textit{own predicted hard labels} (i.e., 1/0 labels). Yet,…
Towards Quantifying the Preconditioning Effect of Adam
Rudrajit Das, Naman Agarwal, Sujay Sanghavi +1
There is a notable dearth of results characterizing the preconditioning effect of Adam and showing how it may alleviate the curse of ill-conditioning -- an issue plaguing gradient…
In-Context Learning with Transformers: Softmax Attention Adapts to Function Lipschitzness
Liam Collins, Advait Parulekar, Aryan Mokhtari +2
A striking property of transformers is their ability to perform in-context learning (ICL), a machine learning framework in which the learner is presented with a novel context durin…
Understanding the Training Speedup from Sampling with Approximate Losses
Rudrajit Das, Xi Chen, Bertram Ieong +2
It is well known that selecting samples with large losses/gradients can significantly reduce the number of training steps. However, the selection overhead is often too high to yiel…
Understanding Contrastive Representation Learning from Positive Unlabeled (PU) Data
Anish Acharya, Li Jing, Bhargav Bhushanam +4
Pretext Invariant Representation Learning (PIRL) followed by Supervised Fine-Tuning (SFT) has become a standard paradigm for learning with limited labels. We extend this approach t…
Pretrained deep models outperform GBDTs in Learning-To-Rank under label scarcity
Charlie Hou, Kiran Koshy Thekumparampil, Michael Shavlovsky +3
On tabular data, a significant body of literature has shown that current deep learning (DL) models perform at best similarly to Gradient Boosted Decision Trees (GBDTs), while signi…
Early Weight Averaging meets High Learning Rates for LLM Pre-training
Sunny Sanyal, Atula Neerkaje, Jean Kaddour +2
Training Large Language Models (LLMs) incurs significant cost; hence, any strategy that accelerates model convergence is helpful. In this paper, we investigate the ability of a sim…
Understanding Self-Distillation in the Presence of Label Noise
Rudrajit Das, Sujay Sanghavi
Self-distillation (SD) is the process of first training a \enquote{teacher} model and then using its predictions to train a \enquote{student} model with the \textit{same} architect…
Latent Variable Representation for Reinforcement Learning
Tongzheng Ren, Chenjun Xiao, Tianjun Zhang +5
Deep latent variable models have achieved significant empirical successes in model-based reinforcement learning (RL) due to their expressiveness in modeling complex transition dyna…
Toward Understanding Privileged Features Distillation in Learning-to-Rank
Shuo Yang, Sujay Sanghavi, Holakou Rahmanian +2
In learning-to-rank problems, a privileged feature is one that is available during model training, but not available at test time. Such features naturally arise in merchandised rec…
Positive Unlabeled Contrastive Learning
Anish Acharya, Sujay Sanghavi, Li Jing +4
Self-supervised pretraining on unlabeled data followed by supervised fine-tuning on labeled data is a popular paradigm for learning from limited labeled examples. We extend this pa…
Sample Efficiency of Data Augmentation Consistency Regularization
Shuo Yang, Yijun Dong, Rachel Ward +3
Data augmentation is popular in the training of large neural networks; currently, however, there is no clear theoretical comparison between different algorithmic choices on how to…
Combinatorial Bandits without Total Order for Arms
Shuo Yang, Tongzheng Ren, Inderjit S. Dhillon +1
We consider the combinatorial bandits problem, where at each time step, the online learner selects a size- subset from the arms set , where $\left|\mathcal{A}\r…
Linear Bandit Algorithms with Sublinear Time Complexity
Shuo Yang, Tongzheng Ren, Sanjay Shakkottai +3
We propose two linear bandits algorithms with per-step complexity sublinear in the number of arms . The algorithms are designed for applications where the arm set is extremely l…
On the Benefits of Multiple Gossip Steps in Communication-Constrained Decentralized Optimization
Abolfazl Hashemi, Anish Acharya, Rudrajit Das +3
In decentralized optimization, it is common algorithmic practice to have nodes interleave (local) gradient descent iterations with gossip (i.e. averaging over the network) steps. M…
Extreme Multi-label Classification from Aggregated Labels
Yanyao Shen, Hsiang-fu Yu, Sujay Sanghavi +1
Extreme multi-label classification (XMC) is the problem of finding the relevant labels for an input, from a very large universe of possible labels. We consider XMC in the setting w…
Interaction Hard Thresholding: Consistent Sparse Quadratic Regression in Sub-quadratic Time and Space
Shuo Yang, Yanyao Shen, Sujay Sanghavi
Quadratic regression involves modeling the response as a (generalized) linear function of not only the features but also of quadratic terms . The inclusio…
Learning Distributions Generated by One-Layer ReLU Networks
Shanshan Wu, Alexandros G. Dimakis, Sujay Sanghavi
We consider the problem of estimating the parameters of a -dimensional rectified Gaussian distribution from i.i.d. samples. A rectified Gaussian distribution is defined by passi…
Blocking Bandits
Soumya Basu, Rajat Sen, Sujay Sanghavi +1
We consider a novel stochastic multi-armed bandit setting, where playing an arm makes it unavailable for a fixed number of time slots thereafter. This models situations where reusi…
PruneTrain: Fast Neural Network Training by Dynamic Sparse Model Reconfiguration
Sangkug Lym, Esha Choukse, Siavash Zangeneh +3
State-of-the-art convolutional neural networks (CNNs) used in vision applications have large models with numerous weights. Training these models is very compute- and memory-resourc…
Clustering Partially Observed Graphs via Convex Optimization
Yudong Chen, Ali Jalali, Sujay Sanghavi +1
This paper considers the problem of clustering a partially observed unweighted graph---i.e., one where for some node pairs we know there is an edge between them, for some others we…