From the 1 of 15 linked papers with an AI index.
204 citations · 389 across the 11 of their papers we have counts for
15 papers
Understanding Contrastive Learning Requires Incorporating Inductive Biases
Nikunj Saunshi, Jordan Ash, Surbhi Goel +5
Contrastive learning is a popular form of self-supervised learning that encourages augmentations (views) of the same input to have more similar representations compared to augmenta…
Rip van Winkle's Razor: A Simple Estimate of Overfit to Test Data
Sanjeev Arora, Yi Zhang
Traditional statistics forbids use of test data (a.k.a. holdout data) during training. Dwork et al. 2015 pointed out that current practices in machine learning, whereby researchers…
On the Validity of Modeling SGD with Stochastic Differential Equations (SDEs)
Zhiyuan Li, Sadhika Malladi, Sanjeev Arora
It is generally recognized that finite learning rate (LR), in contrast to infinitesimal LR, is important for good generalization in real-life deep nets. Most attempted explanations…
TextHide: Tackling Data Privacy in Language Understanding Tasks
Yangsibo Huang, Zhao Song, Danqi Chen +2
An unsolved challenge in distributed or federated learning is to effectively mitigate privacy risks without slowing down training or reducing accuracy. In this paper, we propose Te…
Reconciling Modern Deep Learning with Traditional Optimization Analyses: The Intrinsic Learning Rate
Zhiyuan Li, Kaifeng Lyu, Sanjeev Arora
The paper proposes an 'intrinsic learning rate' that combines the usual learning rate and weight decay to model the training dynamics of normalized deep networks via stochastic dif…
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Nikunj Saunshi, Sadhika Malladi, Sanjeev Arora
Autoregressive language models, pretrained using large text corpora to do well on next word prediction, have been successful at solving many downstream tasks, even with zero-shot u…