8 citations · 24 across the 16 of their papers we have counts for
19 papers
MoRE: Scaling mixture of experts with hardware-aware low-rank routing
Honam Wong, Surbhi Goel, Enric Boix-Adserà
Mixture-of-Experts (MoE) layers are central to frontier language models, and recent architectures push toward more and smaller experts. In this regime, the standard linear router b…
Logical subspace in LLMs
Hope Kean, Enric Boix-Adsera
Recent work has identified a human brain network specialized for abstract formal reasoning (Kean et al., 2025). Does the same hold true in language models? To answer this question,…
Revenge of Monosemanticity: Neuron Specialization as a New Form of Feature Learning in MLPs
Amirhesam Abedsoltan, Enric Boix-Adsera, Fivos Kalogiannis +1
Understanding how neural networks learn and organize features is central to understanding their behavior. Much existing theory of feature learning has focused on the emergence of a…
Model Hypnosis: Strong control of AI via additive subliminal effects
Enric Boix-Adsera, Benedict Tessler
We demonstrate that AI models are broadly susceptible to a phenomenon we call model hypnosis, in which individually weak and seemingly irrelevant cues in the prompt can be systemat…
There Will Be a Scientific Theory of Deep Learning
Jamie Simon, Daniel Kunin, Alexander Atanasov +11
In this paper, we make the case that a scientific theory of deep learning is emerging. By this we mean a theory which characterizes important properties and statistics of the train…
The Features at Convergence Theorem: a first-principles alternative to the Neural Feature Ansatz for how networks learn representations
Enric Boix-Adsera, Neil Mallinar, James B. Simon +1
It is a central challenge in deep learning to understand how neural networks learn representations. A leading approach is the Neural Feature Ansatz (NFA) (Radhakrishnan et al. 2024…
Let Me Think! A Long Chain-of-Thought Can Be Worth Exponentially Many Short Ones
Parsa Mirtaheri, Ezra Edelman, Samy Jelassi +2
Inference-time computation has emerged as a promising scaling axis for improving large language model reasoning. However, despite yielding impressive performance, the optimal alloc…
Toward universal steering and monitoring of AI models
Daniel Beaglehole, Adityanarayanan Radhakrishnan, Enric Boix-Adserà +1
Modern AI models contain much of human knowledge, yet understanding of their internal representation of this knowledge remains elusive. Characterizing the structure and properties…
On the inductive bias of infinite-depth ResNets and the bottleneck rank
Enric Boix-Adsera
We compute the minimum-norm weights of a deep linear ResNet, and find that the inductive bias of this architecture lies between minimizing nuclear norm and rank. This implies that,…
Towards a theory of model distillation
Enric Boix-Adsera
Distillation is the task of replacing a complicated machine learning model with a simpler model that approximates the original [BCNM06,HVD15]. Despite many practical applications,…
Prompts have evil twins
Rimon Melamed, Lucas H. McCabe, Tanay Wakhare +3
We discover that many natural-language prompts can be replaced by corresponding prompts that are unintelligible to humans but that provably elicit similar behavior in language mode…
When can transformers reason with abstract symbols?
Enric Boix-Adsera, Omid Saremi, Emmanuel Abbe +3
We investigate the capabilities of transformer models on relational reasoning tasks. In these tasks, models are trained on a set of strings encoding abstract relations, and are the…
Transformers learn through gradual rank increase
Enric Boix-Adsera, Etai Littwin, Emmanuel Abbe +2
We identify incremental learning dynamics in transformers, where the difference between trained and initial weights progressively increases in rank. We rigorously prove this occurs…
Tight conditions for when the NTK approximation is valid
Enric Boix-Adsera, Etai Littwin
We study when the neural tangent kernel (NTK) approximation is valid for training a model with the square loss. In the lazy training setting of Chizat et al. 2019, we show that res…
SGD learning on neural networks: leap complexity and saddle-to-saddle dynamics
Emmanuel Abbe, Enric Boix-Adsera, Theodor Misiakiewicz
We investigate the time complexity of SGD learning on fully-connected neural networks with isotropic data. We put forward a complexity measure -- the leap -- which measures how "hi…
On the non-universality of deep learning: quantifying the cost of symmetry
Emmanuel Abbe, Enric Boix-Adsera
We prove limitations on what neural networks trained by noisy gradient descent (GD) can efficiently learn. Our results apply whenever GD training is equivariant, which holds for ma…
The merged-staircase property: a necessary and nearly sufficient condition for SGD learning of sparse functions on two-layer neural networks
Emmanuel Abbe, Enric Boix-Adsera, Theodor Misiakiewicz
It is currently known how to characterize functions that neural networks can learn with SGD for two extremal parameterizations: neural networks in the linear regime, and neural net…
The staircase property: How hierarchical structure can guide deep learning
Emmanuel Abbe, Enric Boix-Adsera, Matthew Brennan +2
This paper identifies a structural property of data distributions that enables deep neural networks to learn hierarchically. We define the "staircase" property for functions over t…
Approximating the Determinant of Well-Conditioned Matrices by Shallow Circuits
Enric Boix-Adserà, Lior Eldar, Saeed Mehraban
The determinant can be computed by classical circuits of depth , and therefore it can also be computed in classical space . Recent progress by Ta-Shma [Ta…