4 papers · 1 filter
The Multiscale Single-Index Model: A Stylized Model for Hierarchical Feature Learning
Joan Bruna
We consider the Multiscale Single-Index Model (MSIM), first introduced in \cite{oymak2021learning}, as a stylized model for hierarchical learning with \emph{scale separation}. Each…
Lost in Tokenization: Fundamental Trade-offs in Graph Tokenization for Transformers
Maya Bechler-Speicher, Gilad Yehudai, Gil Harari +3
Transformers have become a central architecture for graph learning, but their application to graphs requires first choosing a tokenization: a graph-to-token map that determines whi…
Uniform-in-Time Weak Propagation-of-Chaos in Shallow Neural Networks
Margalit Glasgow, Joan Bruna
We consider one-hidden layer neural networks trained in the feature-learning regime using gradient descent, and relate the output of the finite-width network to its…
Geometric Factual Recall in Transformers
Shauli Ravfogel, Gilad Yehudai, Joan Bruna +1
How do transformer language models memorize factual associations? A common view casts internal weight matrices as associative memories over pairs of embeddings, requiring parameter…