1 citations · 1 across the 16 of their papers we have counts for
20 papers
High-Dimensional Asymptotics and Dataset Selection for Private Transfer Learning
Filip Kovačević, Edwige Cyffers, Stefano Sarao Mannelli +1
To commit to buying external data or participate in collaborative learning, one must decide whether the additional data will improve prediction enough to justify the cost. This com…
Grokking through the Lens of Minimum-Norm Interpolation
Gil Kur, Ileana Rugina, Clémentine Carla Juliette Dominé +1
Grokking shows that fitting the training data and learning the underlying signal can occur at very different stages. However, existing theories offer limited quantitative insight i…
-JEPA Spectral Anti-Collapse Regularization for Self-Supervised Learning
Berker Demirel, Clémentine Dominé, Valentino Maiorca +3
Joint-embedding self-supervised learning typically combines an invariance objective across augmented views with additional mechanisms to prevent representational collapse. These ob…
Optimal Estimation in Orthogonally Invariant Generalized Linear Models: Spectral Initialization and Approximate Message Passing
Yihan Zhang, Hong Chang Ji, Ramji Venkataramanan +1
We consider the problem of parameter estimation from a generalized linear model with a random design matrix that is orthogonally invariant in law. Such a model allows the design ha…
A Theory of How Pretraining Shapes Inductive Bias in Fine-Tuning
Nicolas Anguita, Francesco Locatello, Andrew M. Saxe +4
Pretraining and fine-tuning are central stages in modern machine learning systems. In practice, feature learning plays an important role across both stages: deep neural networks le…
Full-Batch Gradient Descent Outperforms One-Pass SGD: Sample Complexity Separation in Single-Index Learning
Filip Kovačević, Hong Chang Ji, Denny Wu +2
It is folklore that reusing training data more than once can improve the statistical efficiency of gradient-based learning. While this phenomenon has been extensively studied in li…
High-dimensional Analysis of Synthetic Data Selection
Parham Rezaei, Filip Kovacevic, Francesco Locatello +1
Despite the progress in the development of generative models, their usefulness in creating synthetic data that improve prediction performance of classifiers has been put into quest…
Optimal Regularization for Performative Learning
Edwige Cyffers, Alireza Mirrokni, Marco Mondelli
In performative learning, the data distribution reacts to the deployed model - for example, because strategic users adapt their features to game it - which creates a more complex d…
Attention with Trained Embeddings Provably Selects Important Tokens
Diyuan Wu, Aleksandr Shevchenko, Samet Oymak +1
Token embeddings play a crucial role in language modeling but, despite this practical relevance, their theoretical understanding remains limited. Our paper addresses the gap by cha…
Neural Collapse is Globally Optimal in Deep Regularized ResNets and Transformers
Peter Súkeník, Christoph H. Lampert, Marco Mondelli
The empirical emergence of neural collapse -- a surprising symmetry in the feature representations of the training data in the penultimate layer of deep neural networks -- has spur…
Test-Time Training Provably Improves Transformers as In-context Learners
Halil Alperen Gozeten, M. Emrullah Ildiz, Xuechen Zhang +3
Test-time training (TTT) methods explicitly update the weights of a model to adapt to the specific test instance, and they have found success in a variety of settings, including mo…
Neural Collapse Beyond the Unconstrained Features Model: Landscape, Dynamics, and Generalization in the Mean-Field Regime
Diyuan Wu, Marco Mondelli
Neural Collapse is a phenomenon where the last-layer representations of a well-trained neural network converge to a highly structured geometry. In this paper, we focus on its first…
Spectral Estimators for Multi-Index Models: Precise Asymptotics and Optimal Weak Recovery
Filip Kovačević, Yihan Zhang, Marco Mondelli
Multi-index models provide a popular framework to investigate the learnability of functions with low-dimensional structure and, also due to their connections with neural networks,…
Spurious Correlations in High Dimensional Regression: The Roles of Regularization, Simplicity Bias and Over-Parameterization
Simone Bombari, Marco Mondelli
Learning models have been shown to rely on spurious correlations between non-predictive features and the associated labels in the training data, with negative implications on robus…
Wide Neural Networks Trained with Weight Decay Provably Exhibit Neural Collapse
Arthur Jacot, Peter Súkeník, Zihan Wang +1
Deep neural networks (DNNs) at convergence consistently represent the training data in the last layer via a highly symmetric geometric structure referred to as neural collapse. Thi…
High-dimensional Analysis of Knowledge Distillation: Weak-to-Strong Generalization and Scaling Laws
M. Emrullah Ildiz, Halil Alperen Gozeten, Ege Onur Taga +2
A growing number of machine learning scenarios rely on knowledge distillation where one uses the output of a surrogate model as labels to supervise the training of a target model.…
Information limits and Thouless-Anderson-Palmer equations for spiked matrix models with structured noise
Jean Barbier, Francesco Camilli, Marco Mondelli +1
We consider a prototypical problem of Bayesian inference for a structured spiked model: a low-rank signal is corrupted by additive noise. While both information-theoretic and algor…
Neural Collapse versus Low-rank Bias: Is Deep Neural Collapse Really Optimal?
Peter Súkeník, Marco Mondelli, Christoph Lampert
Deep neural networks (DNNs) exhibit a surprising structure in their final layer known as neural collapse (NC), and a growing body of works has currently investigated the propagatio…
Matrix Denoising with Doubly Heteroscedastic Noise: Fundamental Limits and Optimal Spectral Methods
Yihan Zhang, Marco Mondelli
We study the matrix denoising problem of estimating the singular vectors of a rank- signal corrupted by noise with both column and row correlations. Existing works are either un…
Spectral Estimators for Structured Generalized Linear Models via Approximate Message Passing
Yihan Zhang, Hong Chang Ji, Ramji Venkataramanan +1
We consider the problem of parameter estimation in a high-dimensional generalized linear model. Spectral methods obtained via the principal eigenvector of a suitable data-dependent…