50 citations · 223 across the 27 of their papers we have counts for
28 papers
Sharp Capacity Scaling of Spectral Optimizers in Learning Associative Memory
Juno Kim, Eshaan Nichani, Denny Wu +2
Spectral optimizers such as Muon have recently shown strong empirical performance in large-scale language model training, but the source and extent of their advantage remain poorly…
Emergence of Linear Truth Encodings in Language Models
Shauli Ravfogel, Gilad Yehudai, Tal Linzen +2
Recent probing studies reveal that large language models exhibit linear subspaces that separate true from false statements, yet the mechanism behind their emergence is unclear. We…
Distributional Associations vs In-Context Reasoning: A Study of Feed-forward and Attention Layers
Lei Chen, Joan Bruna, Alberto Bietti
Large language models have been successful at tasks involving basic forms of in-context reasoning, such as generating coherent language, as well as storing vast amounts of knowledg…
Level Set Teleportation: An Optimization Perspective
Aaron Mishkin, Alberto Bietti, Robert M. Gower
We study level set teleportation, an optimization routine which tries to accelerate gradient descent (GD) by maximizing the gradient norm over a level set of the objective. While t…
Learning Associative Memories with Gradient Descent
Vivien Cabannes, Berfin Simsek, Alberto Bietti
This work focuses on the training dynamics of one associative memory module storing outer products of token embeddings. We reduce this problem to the study of a system of particles…
On Learning Gaussian Multi-index Models with Gradient Flow
Alberto Bietti, Joan Bruna, Loucas Pillaud-Vivien
We study gradient flow on the multi-index regression problem for high-dimensional Gaussian data. Multi-index functions consist of a composition of an unknown low-rank linear projec…
Scaling Laws for Associative Memories
Vivien Cabannes, Elvis Dohmatob, Alberto Bietti
Learning arguably involves the discovery and memorization of abstract rules. The aim of this paper is to study associative memory mechanisms. Our model is based on high-dimensional…
Birth of a Transformer: A Memory Viewpoint
Alberto Bietti, Vivien Cabannes, Diane Bouchacourt +2
Large language models based on transformers have achieved great empirical successes. However, as they are deployed more widely, there is a growing need to better understand their i…
The SSL Interplay: Augmentations, Inductive Bias, and Generalization
Vivien Cabannes, Bobak T. Kiani, Randall Balestriero +2
Self-supervised learning (SSL) has emerged as a powerful framework to learn representations from raw data without supervision. Yet in practice, engineers face issues such as instab…
On minimal variations for unsupervised representation learning
Vivien Cabannes, Alberto Bietti, Randall Balestriero
Unsupervised representation learning aims at describing raw data efficiently to solve various downstream tasks. It has been approached with many techniques, such as manifold learni…
Learning Single-Index Models with Shallow Neural Networks
Alberto Bietti, Joan Bruna, Clayton Sanford +1
Single-index models are a class of functions given by an unknown univariate ``link'' function applied to an unknown one-dimensional projection of the input. These models are partic…
When does return-conditioned supervised learning work for offline reinforcement learning?
David Brandfonbrener, Alberto Bietti, Jacob Buckman +2
Several recent works have proposed a class of algorithms for the offline reinforcement learning (RL) problem that we will refer to as return-conditioned supervised learning (RCSL).…
On the (Non-)Robustness of Two-Layer Neural Networks in Different Learning Regimes
Elvis Dohmatob, Alberto Bietti
Neural networks are known to be highly sensitive to adversarial examples. These may arise due to different factors, such as random initialization, or spurious correlations in the l…
Efficient Kernel UCB for Contextual Bandits
Houssam Zenati, Alberto Bietti, Eustache Diemert +3
In this paper, we tackle the computational efficiency of kernelized UCB algorithms in contextual bandits. While standard methods require a O(CT^3) complexity where T is the horizon…
Personalization Improves Privacy-Accuracy Tradeoffs in Federated Learning
Alberto Bietti, Chen-Yu Wei, Miroslav Dudík +2
Large-scale machine learning systems often involve data distributed across a collection of users. Federated learning algorithms leverage this structure by communicating model updat…
Dual Training of Energy-Based Models with Overparametrized Shallow Neural Networks
Carles Domingo-Enrich, Alberto Bietti, Marylou Gabrié +2
Energy-based models (EBMs) are generative models that are usually trained via maximum likelihood estimation. This approach becomes challenging in generic situations where the train…
On the Sample Complexity of Learning under Invariance and Geometric Stability
Alberto Bietti, Luca Venturi, Joan Bruna
Many supervised learning problems involve high-dimensional data such as images, text, or graphs. In order to make efficient use of data, it is often useful to leverage certain geom…
On the Universality of Graph Neural Networks on Large Random Graphs
Nicolas Keriven, Alberto Bietti, Samuel Vaiter
We study the approximation power of Graph Neural Networks (GNNs) on latent position random graphs. In the large graph limit, GNNs are known to converge to certain "continuous" mode…
On Energy-Based Models with Overparametrized Shallow Neural Networks
Carles Domingo-Enrich, Alberto Bietti, Eric Vanden-Eijnden +1
Energy-based models (EBMs) are a simple yet powerful framework for generative modeling. They are based on a trainable energy function which defines an associated Gibbs measure, and…
Approximation and Learning with Deep Convolutional Models: a Kernel Perspective
Alberto Bietti
The empirical success of deep convolutional networks on tasks involving high-dimensional data such as images or audio suggests that they can efficiently approximate certain functio…
Deep Equals Shallow for ReLU Networks in Kernel Regimes
Alberto Bietti, Francis Bach
Deep networks are often considered to be more expressive than shallow ones in terms of approximation. Indeed, certain functions can be approximated by deep networks provably more e…
Convergence and Stability of Graph Convolutional Networks on Large Random Graphs
Nicolas Keriven, Alberto Bietti, Samuel Vaiter
We study properties of Graph Convolutional Networks (GCNs) by analyzing their behavior on standard models of random graphs, where nodes are represented by random latent variables a…
Counterfactual Learning of Stochastic Policies with Continuous Actions
Houssam Zenati, Alberto Bietti, Matthieu Martin +3
Counterfactual reasoning from logged data has become increasingly important for many applications such as web advertising or healthcare. In this paper, we address the problem of le…
On the Inductive Bias of Neural Tangent Kernels
Alberto Bietti, Julien Mairal
State-of-the-art neural networks are heavily over-parameterized, making the optimization algorithm a crucial ingredient for learning predictive models with good generalization prop…