10 papers
Flatness and Generalization: Learning Multi-Index Models with Homogeneous Neural Networks
Harsh Vardhan, Hossein Taheri, Arya Mazumdar
A common heuristic used to explain the generalization of first-order gradient methods on non-convex neural networks is that "flat interpolators generalize well" (Hochreiter and Sch…
Improving Selective Classification with Pairwise Queries for Binary Classification
Harsh Vardhan, Sunav Choudhary, Natwar Modani +1
In selective classification, a model predicts the labels of data samples where it is confident, and abstains from predicting labels for samples on which it is not confident. The re…
Investigation into In-Context Learning Capabilities of Transformers
Rushil Chandrupatla, Leo Bangayan, Sebastian Leng
Transformers have demonstrated a strong ability for in-context learning (ICL), enabling models to solve previously unseen tasks using only example input output pairs provided at in…
Collaborative Compressors in Distributed Mean Estimation with Limited Communication Budget
Harsh Vardhan, Arya Mazumdar
Distributed high dimensional mean estimation is a common aggregation routine used often in distributed optimization methods. Most of these applications call for a communication-con…
On the Theory of Continual Learning with Gradient Descent for Neural Networks
Hossein Taheri, Avishek Ghosh, Arya Mazumdar
Continual learning, the ability of a model to adapt to an ongoing sequence of tasks without forgetting earlier ones, is a central goal of artificial intelligence. To better underst…
LocalKMeans: Convergence of Lloyd's Algorithm with Distributed Local Iterations
Harsh Vardhan, Heng Zhu, Avishek Ghosh +1
In this paper, we analyze the classical -means alternating-minimization algorithm, also known as Lloyd's algorithm (Lloyd, 1956), for a mixture of Gaussians in a data-distribute…
Learning and Generalization with Mixture Data
Harsh Vardhan, Avishek Ghosh, Arya Mazumdar
In many, if not most, machine learning applications the training data is naturally heterogeneous (e.g. federated learning, adversarial attacks and domain adaptation in neural net t…
Client Selection in Federated Learning with Data Heterogeneity and Network Latencies
Harsh Vardhan, Xiaofan Yu, Tajana Rosing +1
Federated learning (FL) is a distributed machine learning paradigm where multiple clients conduct local training based on their private data, then the updated models are sent to a…
Distributed Hybrid Sketching for -Embeddings
Neophytos Charalambides, Arya Mazumdar
Linear algebraic operations are ubiquitous in engineering applications, and arise often in a variety of fields including statistical signal processing and machine learning. With co…
Effectiveness of Distributed Gradient Descent with Local Steps for Overparameterized Models
Heng Zhu, Harsh Vardhan, Arya Mazumdar
In distributed training of machine learning models, gradient descent with local iterative steps, commonly known as Local (Stochastic) Gradient Descent (Local-(S)GD) or Federated av…