26 citations · 137 across the 41 of their papers we have counts for
11 papers · 1 filter
Average gradient outer product as a mechanism for deep neural collapse
Daniel Beaglehole, Peter Súkeník, Marco Mondelli +1
Deep Neural Collapse (DNC) refers to the surprisingly rigid structure of the data representations in the final layers of Deep Neural Networks (DNNs). Though the phenomenon has been…
Deep Neural Collapse Is Provably Optimal for the Deep Unconstrained Features Model
Peter Súkeník, Marco Mondelli, Christoph Lampert
Neural collapse (NC) refers to the surprising structure of the last layer of deep neural networks in the terminal phase of gradient descent training. Recently, an increasing amount…
Improved Convergence of Score-Based Diffusion Models via Prediction-Correction
Francesco Pedrotti, Jan Maas, Marco Mondelli
Score-based generative models (SGMs) are powerful tools to sample from complex data distributions. Their underlying idea is to (i) run a forward process for time by adding no…
Fundamental Limits of Two-layer Autoencoders, and Achieving Them with Gradient Methods
Alexander Shevchenko, Kevin Kögler, Hamed Hassani +1
Autoencoders are a popular model in many branches of machine learning and lossy data compression. However, their fundamental limits, the performance of gradient methods and the fea…
Finite Sample Identification of Wide Shallow Neural Networks with Biases
Massimo Fornasier, Timo Klock, Marco Mondelli +1
Artificial neural networks are functions depending on a finite number of parameters typically encoded as weights and biases. The identification of the parameters of the network fro…
Mean-field analysis for heavy ball methods: Dropout-stability, connectivity, and global convergence
Diyuan Wu, Vyacheslav Kungurtsev, Marco Mondelli
The stochastic heavy ball method (SHB), also known as stochastic gradient descent (SGD) with Polyak's momentum, is widely used in training neural networks. However, despite the rem…
Mean-field Analysis of Piecewise Linear Solutions for Wide ReLU Networks
Alexander Shevchenko, Vyacheslav Kungurtsev, Marco Mondelli
Understanding the properties of neural networks trained via stochastic gradient descent (SGD) is at the heart of the theory of deep learning. In this work, we take a mean-field vie…
When Are Solutions Connected in Deep Networks?
Quynh Nguyen, Pierre Brechet, Marco Mondelli
The question of how and why the phenomenon of mode connectivity occurs in training deep neural networks has gained remarkable attention in the research community. From a theoretica…
Global Convergence of Deep Networks with One Wide Layer Followed by Pyramidal Topology
Quynh Nguyen, Marco Mondelli
Recent works have shown that gradient descent can find a global minimum for over-parameterized neural networks where the widths of all the hidden layers scale polynomially with …
Landscape Connectivity and Dropout Stability of SGD Solutions for Over-parameterized Neural Networks
Alexander Shevchenko, Marco Mondelli
The optimization of multilayer neural networks typically leads to a solution with zero training error, yet the landscape can exhibit spurious local minima and the minima can be dis…
On the Connection Between Learning Two-Layers Neural Networks and Tensor Decomposition
Marco Mondelli, Andrea Montanari
We establish connections between the problem of learning a two-layer neural network and tensor decomposition. We consider a model with feature vectors $\boldsymbol x \in \mathbb R^…