50 citations · 223 across the 27 of their papers we have counts for
10 papers · 1 filter
Sharp Capacity Scaling of Spectral Optimizers in Learning Associative Memory
Juno Kim, Eshaan Nichani, Denny Wu +2
Spectral optimizers such as Muon have recently shown strong empirical performance in large-scale language model training, but the source and extent of their advantage remain poorly…
Distributional Associations vs In-Context Reasoning: A Study of Feed-forward and Attention Layers
Lei Chen, Joan Bruna, Alberto Bietti
Large language models have been successful at tasks involving basic forms of in-context reasoning, such as generating coherent language, as well as storing vast amounts of knowledg…
Level Set Teleportation: An Optimization Perspective
Aaron Mishkin, Alberto Bietti, Robert M. Gower
We study level set teleportation, an optimization routine which tries to accelerate gradient descent (GD) by maximizing the gradient norm over a level set of the objective. While t…
Learning Associative Memories with Gradient Descent
Vivien Cabannes, Berfin Simsek, Alberto Bietti
This work focuses on the training dynamics of one associative memory module storing outer products of token embeddings. We reduce this problem to the study of a system of particles…
On minimal variations for unsupervised representation learning
Vivien Cabannes, Alberto Bietti, Randall Balestriero
Unsupervised representation learning aims at describing raw data efficiently to solve various downstream tasks. It has been approached with many techniques, such as manifold learni…
Learning Single-Index Models with Shallow Neural Networks
Alberto Bietti, Joan Bruna, Clayton Sanford +1
Single-index models are a class of functions given by an unknown univariate ``link'' function applied to an unknown one-dimensional projection of the input. These models are partic…
When does return-conditioned supervised learning work for offline reinforcement learning?
David Brandfonbrener, Alberto Bietti, Jacob Buckman +2
Several recent works have proposed a class of algorithms for the offline reinforcement learning (RL) problem that we will refer to as return-conditioned supervised learning (RCSL).…
Efficient Kernel UCB for Contextual Bandits
Houssam Zenati, Alberto Bietti, Eustache Diemert +3
In this paper, we tackle the computational efficiency of kernelized UCB algorithms in contextual bandits. While standard methods require a O(CT^3) complexity where T is the horizon…
Dual Training of Energy-Based Models with Overparametrized Shallow Neural Networks
Carles Domingo-Enrich, Alberto Bietti, Marylou Gabrié +2
Energy-based models (EBMs) are generative models that are usually trained via maximum likelihood estimation. This approach becomes challenging in generic situations where the train…
On Energy-Based Models with Overparametrized Shallow Neural Networks
Carles Domingo-Enrich, Alberto Bietti, Eric Vanden-Eijnden +1
Energy-based models (EBMs) are a simple yet powerful framework for generative modeling. They are based on a trainable energy function which defines an associated Gibbs measure, and…