970 citations · 3k across the 88 of their papers we have counts for
90 papers
A projection-based framework for gradient-free and parallel learning
Andreas Bergmeister, Manish Krishan Lal, Stefanie Jegelka +1
We present a feasibility-seeking approach to neural network training. This mathematical optimization framework is distinct from conventional gradient-based loss minimization and us…
On the Role of Depth and Looping for In-Context Learning with Task Diversity
Khashayar Gatmiry, Nikunj Saunshi, Sashank J. Reddi +2
The intriguing in-context learning (ICL) abilities of deep Transformer models have lately garnered significant attention. By studying in-context linear regression on unimodal Gauss…
Computing Optimal Regularizers for Online Linear Optimization
Khashayar Gatmiry, Jon Schneider, Stefanie Jegelka
Follow-the-Regularized-Leader (FTRL) algorithms are a popular class of learning algorithms for online linear optimization (OLO) that guarantee sub-linear regret, but the choice of…
Simplicity Bias via Global Convergence of Sharpness Minimization
Khashayar Gatmiry, Zhiyuan Li, Sashank J. Reddi +1
The remarkable generalization ability of neural networks is usually attributed to the implicit bias of SGD, which often yields models with lower complexity using simpler (e.g. line…
Can Looped Transformers Learn to Implement Multi-step Gradient Descent for In-context Learning?
Khashayar Gatmiry, Nikunj Saunshi, Sashank J. Reddi +2
The remarkable capability of Transformers to do reasoning and few-shot learning, without any fine-tuning, is widely conjectured to stem from their ability to implicitly simulate a…
A Universal Class of Sharpness-Aware Minimization Algorithms
Behrooz Tahmasebi, Ashkan Soleymani, Dara Bahri +2
Recently, there has been a surge in interest in developing optimization algorithms for overparameterized models as achieving generalization is believed to require algorithms with s…
A Poincaré Inequality and Consistency Results for Signal Sampling on Large Graphs
Thien Le, Luana Ruiz, Stefanie Jegelka
Large-scale graph machine learning is challenging as the complexity of learning models scales with the graph size. Subsampling the graph is a viable alternative, but sampling on gr…
Sample Complexity Bounds for Estimating Probability Divergences under Invariances
Behrooz Tahmasebi, Stefanie Jegelka
Group-invariant probability distributions appear in many data-generative models in machine learning, such as graphs, point clouds, and images. In practice, one often needs to estim…
Structuring Representation Geometry with Rotationally Equivariant Contrastive Learning
Sharut Gupta, Joshua Robinson, Derek Lim +2
Self-supervised learning converts raw perceptual data such as images to a compact space where simple Euclidean distances measure meaningful variations in data. In this paper, we ex…
The Inductive Bias of Flatness Regularization for Deep Matrix Factorization
Khashayar Gatmiry, Zhiyuan Li, Ching-Yao Chuang +3
Recent works on over-parameterized neural networks have shown that the stochasticity in optimizers has the implicit regularization effect of minimizing the sharpness of the loss fu…
Limits, approximation and size transferability for GNNs on sparse graphs via graphops
Thien Le, Stefanie Jegelka
Can graph neural networks generalize to graphs that are different from the graphs they were trained on, e.g., in size? In this work, we study this question from a theoretical persp…
The Exact Sample Complexity Gain from Invariances for Kernel Regression
Behrooz Tahmasebi, Stefanie Jegelka
In practice, encoding invariances into models improves sample complexity. In this work, we study this phenomenon from a theoretical perspective. In particular, we provide minimax o…
Debiasing Vision-Language Models via Biased Prompts
Ching-Yao Chuang, Varun Jampani, Yuanzhen Li +2
Machine learning models have been shown to inherit biases from their training datasets. This can be particularly problematic for vision-language foundation models trained on uncura…
Efficiently predicting high resolution mass spectra with graph neural networks
Michael Murphy, Stefanie Jegelka, Ernest Fraenkel +3
Identifying a small molecule from its mass spectrum is the primary open problem in computational metabolomics. This is typically cast as information retrieval: an unknown spectrum…
Near-Optimal Algorithms for Group Distributionally Robust Optimization and Beyond
Tasuku Soma, Khashayar Gatmiry, Sharut Gupta +1
Distributionally robust optimization (DRO) can improve the robustness and fairness of learning methods. In this paper, we devise stochastic algorithms for a class of DRO problems i…
Tree Mover's Distance: Bridging Graph Metrics and Stability of Graph Neural Networks
Ching-Yao Chuang, Stefanie Jegelka
Understanding generalization and robustness of machine learning models fundamentally relies on assuming an appropriate metric on the data space. Identifying such a metric is partic…
InfoOT: Information Maximizing Optimal Transport
Ching-Yao Chuang, Stefanie Jegelka, David Alvarez-Melis
Optimal transport aligns samples across distributions by minimizing the transportation cost between them, e.g., the geometric distances. Yet, it ignores coherence structure in the…
On the generalization of learning algorithms that do not converge
Nisha Chandramoorthy, Andreas Loukas, Khashayar Gatmiry +1
Generalization analyses of deep learning typically assume that the training converges to a fixed point. But, recent results indicate that in practice, the weights of deep neural ne…
Neural Set Function Extensions: Learning with Discrete Functions in High Dimensions
Nikolaos Karalias, Joshua Robinson, Andreas Loukas +1
Integrating functions on discrete domains into neural networks is key to developing their capability to reason about discrete objects. But, discrete domains are (1) not naturally a…
Theory of Graph Neural Networks: Representation and Learning
Stefanie Jegelka
Graph Neural Networks (GNNs), neural network architectures targeted to learning representations of graphs, have become a popular learning model for prediction tasks on nodes, graph…
Sign and Basis Invariant Networks for Spectral Graph Representation Learning
Derek Lim, Joshua Robinson, Lingxiao Zhao +4
We introduce SignNet and BasisNet -- new neural architectures that are invariant to two key symmetries displayed by eigenvectors: (i) sign flips, since if is an eigenvector the…
Robust Contrastive Learning against Noisy Views
Ching-Yao Chuang, R Devon Hjelm, Xin Wang +5
Contrastive learning relies on an assumption that positive pairs contain related views, e.g., patches of an image or co-occurring multimodal signals of a video, that share certain…
Training invariances and the low-rank phenomenon: beyond linear networks
Thien Le, Stefanie Jegelka
The implicit bias induced by the training of neural networks has become a topic of rigorous study. In the limit of gradient flow and gradient descent with appropriate step size, it…
Scaling up Continuous-Time Markov Chains Helps Resolve Underspecification
Alkis Gotovos, Rebekka Burkholz, John Quackenbush +1
Modeling the time evolution of discrete sets of items (e.g., genetic mutations) is a fundamental problem in many biomedical applications. We approach this problem through the lens…