483 citations · 760 across the 48 of their papers we have counts for
50 papers
SGD in Multiclass Logistic Regression: Sequential Learning and Scaling Laws
Konstantinos Christopher Tsiolis, Denny Wu, Christos Thrampoulidis +1
We study the training dynamics of multiclass logistic regression on high-dimensional Gaussian mixture models with a large number of classes and establish precise scaling laws gover…
Post-Training with Policy Gradients: Optimality and the Base Model Barrier
Alireza Mousavi-Hosseini, Murat A. Erdogdu
We study post-training linear autoregressive models with outcome and process rewards. Given a context , the model must predict the response …
From Information to Generative Exponent: Learning Rate Induces Phase Transitions in SGD
Konstantinos Christopher Tsiolis, Alireza Mousavi-Hosseini, Murat A. Erdogdu
To understand feature learning dynamics in neural networks, recent theoretical works have focused on gradient-based learning of Gaussian single-index models, where the label is a n…
A Geometric Analysis of PCA
Ayoub El Hanchi, Murat Erdogdu, Chris Maddison
What property of the data distribution determines the excess risk of principal component analysis? In this paper, we provide a precise answer to this question. We establish a centr…
Learning quadratic neural networks in high dimensions: SGD dynamics and scaling laws
Gérard Ben Arous, Murat A. Erdogdu, Nuri Mert Vural +1
We study the optimization and sample complexity of gradient-based training of a two-layer neural network with quadratic activation function in the high-dimensional regime, where th…
Distributional Training Data Attribution: What do Influence Functions Sample?
Bruno Mlodozeniec, Isaac Reid, Sam Power +4
Randomness is an unavoidable part of training deep learning models, yet something that traditional training data attribution algorithms fail to rigorously account for. They ignore…
When Do Transformers Outperform Feedforward and Recurrent Networks? A Statistical Perspective
Alireza Mousavi-Hosseini, Clayton Sanford, Denny Wu +1
Theoretical efforts to prove advantages of Transformers in comparison with classical architectures such as feedforward and recurrent neural networks have mostly focused on represen…
On the Efficiency of ERM in Feature Learning
Ayoub El Hanchi, Chris J. Maddison, Murat A. Erdogdu
Given a collection of feature maps indexed by a set , we study the performance of empirical risk minimization (ERM) on regression problems with square loss over the un…
Robust Feature Learning for Multi-Index Models in High Dimensions
Alireza Mousavi-Hosseini, Adel Javanmard, Murat A. Erdogdu
Recently, there have been numerous studies on feature learning with neural networks, specifically on learning single- and multi-index models where the target is a function of a low…
Learning Multi-Index Models with Neural Networks via Mean-Field Langevin Dynamics
Alireza Mousavi-Hosseini, Denny Wu, Murat A. Erdogdu
We study the problem of learning multi-index models in high-dimensions using a two-layer neural network trained with the mean-field Langevin algorithm. Under mild distributional as…
Minimax Linear Regression under the Quantile Risk
Ayoub El Hanchi, Chris J. Maddison, Murat A. Erdogdu
We study the problem of designing minimax procedures in linear regression under the quantile risk. We start by considering the realizable setting with independent Gaussian noise, w…
Pruning is Optimal for Learning Sparse Features in High-Dimensions
Nuri Mert Vural, Murat A. Erdogdu
While it is commonly observed in practice that pruning networks to a certain level of sparsity can improve the quality of the features, a theoretical explanation of this phenomenon…
A Separation in Heavy-Tailed Sampling: Gaussian vs. Stable Oracles for Proximal Samplers
Ye He, Alireza Mousavi-Hosseini, Krishnakumar Balasubramanian +1
We study the complexity of heavy-tailed sampling and present a separation result in terms of obtaining high-accuracy versus low-accuracy guarantees i.e., samplers that require only…
Sampling from the Mean-Field Stationary Distribution
Yunbum Kook, Matthew S. Zhang, Sinho Chewi +2
We study the complexity of sampling from the stationary distribution of a mean-field SDE, or equivalently, the complexity of minimizing a functional over the space of probability m…
Optimal Excess Risk Bounds for Empirical Risk Minimization on -Norm Linear Regression
Ayoub El Hanchi, Murat A. Erdogdu
We study the performance of empirical risk minimization on the -norm linear regression problem for . We show that, in the realizable case, under no moment ass…
Beyond Labeling Oracles: What does it mean to steal ML models?
Avital Shafran, Ilia Shumailov, Murat A. Erdogdu +1
Model extraction attacks are designed to steal trained models with only query access, as is often provided through APIs that ML-as-a-Service providers offer. Machine Learning (ML)…
Gradient-Based Feature Learning under Structured Data
Alireza Mousavi-Hosseini, Denny Wu, Taiji Suzuki +1
Recent works have demonstrated that the sample complexity of gradient-based learning of single index models, i.e. functions that depend on a 1-dimensional projection of the input d…
Distributional Model Equivalence for Risk-Sensitive Reinforcement Learning
Tyler Kastner, Murat A. Erdogdu, Amir-massoud Farahmand
We consider the problem of learning models for risk-sensitive reinforcement learning. We theoretically demonstrate that proper value equivalence, a method of learning models which…
Towards a Complete Analysis of Langevin Monte Carlo: Beyond Poincaré Inequality
Alireza Mousavi-Hosseini, Tyler Farghly, Ye He +2
Langevin diffusions are rapidly convergent under appropriate functional inequality assumptions. Hence, it is natural to expect that with additional smoothness conditions to handle…
Mean-Square Analysis of Discretized Itô Diffusions for Heavy-tailed Sampling
Ye He, Tyler Farghly, Krishnakumar Balasubramanian +1
We analyze the complexity of sampling from a class of heavy-tailed distributions by discretizing a natural class of Itô diffusions associated with weighted Poincaré inequalities. B…
Improved Discretization Analysis for Underdamped Langevin Monte Carlo
Matthew Zhang, Sinho Chewi, Mufan Bill Li +2
Underdamped Langevin Monte Carlo (ULMC) is an algorithm used to sample from unnormalized densities by leveraging the momentum of a particle moving in a potential well. We provide a…
Generalization Bounds for Stochastic Gradient Descent via Localized -Covers
Sejun Park, Umut Şimşekli, Murat A. Erdogdu
In this paper, we propose a new covering technique localized for the trajectories of SGD. This localization provides an algorithm-specific complexity measured by the covering numbe…
Neural Networks Efficiently Learn Low-Dimensional Representations with SGD
Alireza Mousavi-Hosseini, Sejun Park, Manuela Girotti +2
We study the problem of training a two-layer neural network (NN) of arbitrary width using stochastic gradient descent (SGD) where the input is Gaus…
-DkNN: Out-of-Distribution Detection Through Statistical Testing of Deep Representations
Adam Dziedzic, Stephan Rabanser, Mohammad Yaghini +3
The lack of well-calibrated confidence estimates makes neural networks inadequate in safety-critical domains such as autonomous driving or healthcare. In these settings, having the…