10 papers
There Will Be a Scientific Theory of Deep Learning
Jamie Simon, Daniel Kunin, Alexander Atanasov +11
In this paper, we make the case that a scientific theory of deep learning is emerging. By this we mean a theory which characterizes important properties and statistics of the train…
Saddle-To-Saddle Dynamics in Deep ReLU Networks: Low-Rank Bias in the First Saddle Escape
Ioannis Bantzis, James B. Simon, Arthur Jacot
When a deep ReLU network is initialized with small weights, gradient descent (GD) is at first dominated by the saddle at the origin in parameter space. We study the so-called escap…
Polynomial Speedup in Diffusion Models with the Multilevel Euler-Maruyama Method
Arthur Jacot
We introduce the Multilevel Euler-Maruyama (ML-EM) method compute solutions of SDEs and ODEs using a range of approximators to the drift with increasing accurac…
Deep Learning as a Convex Paradigm of Computation: Minimizing Circuit Size with ResNets
Arthur Jacot
This paper argues that DNNs implement a computational Occam's razor -- finding the `simplest' algorithm that fits the data -- and that this could explain their incredible and wide-…
Hamiltonian Mechanics of Feature Learning: Bottleneck Structure in Leaky ResNets
Arthur Jacot, Alexandre Kaiser
We study Leaky ResNets, which interpolate between ResNets and Fully-Connected nets depending on an 'effective depth' hyper-parameter . In the infinite depth limit, we st…
How DNNs break the Curse of Dimensionality: Compositionality and Symmetry Learning
Arthur Jacot, Seok Hoan Choi, Yuxiao Wen
We show that deep neural networks (DNNs) can efficiently learn any composition of functions with bounded -norm, which allows DNNs to break the curse of dimensionality in way…