10 papers
Flatness and Generalization: Learning Multi-Index Models with Homogeneous Neural Networks
Harsh Vardhan, Hossein Taheri, Arya Mazumdar
A common heuristic used to explain the generalization of first-order gradient methods on non-convex neural networks is that "flat interpolators generalize well" (Hochreiter and Sch…
Improving Selective Classification with Pairwise Queries for Binary Classification
Harsh Vardhan, Sunav Choudhary, Natwar Modani +1
In selective classification, a model predicts the labels of data samples where it is confident, and abstains from predicting labels for samples on which it is not confident. The re…
Investigation into In-Context Learning Capabilities of Transformers
Rushil Chandrupatla, Leo Bangayan, Sebastian Leng
Transformers have demonstrated a strong ability for in-context learning (ICL), enabling models to solve previously unseen tasks using only example input output pairs provided at in…
On the Theory of Continual Learning with Gradient Descent for Neural Networks
Hossein Taheri, Avishek Ghosh, Arya Mazumdar
Continual learning, the ability of a model to adapt to an ongoing sequence of tasks without forgetting earlier ones, is a central goal of artificial intelligence. To better underst…
Effectiveness of Distributed Gradient Descent with Local Steps for Overparameterized Models
Heng Zhu, Harsh Vardhan, Arya Mazumdar
In distributed training of machine learning models, gradient descent with local iterative steps, commonly known as Local (Stochastic) Gradient Descent (Local-(S)GD) or Federated av…
Collaborative Compressors in Distributed Mean Estimation with Limited Communication Budget
Harsh Vardhan, Arya Mazumdar
Distributed high dimensional mean estimation is a common aggregation routine used often in distributed optimization methods. Most of these applications call for a communication-con…