26 citations · 137 across the 41 of their papers we have counts for
53 papers
Privacy for Free in the Overparameterized Regime
Simone Bombari, Marco Mondelli
Differentially private gradient descent (DP-GD) is a popular algorithm to train deep learning models with provable guarantees on the privacy of the training data. In the last decad…
Average gradient outer product as a mechanism for deep neural collapse
Daniel Beaglehole, Peter Súkeník, Marco Mondelli +1
Deep Neural Collapse (DNC) refers to the surprisingly rigid structure of the data representations in the final layers of Deep Neural Networks (DNNs). Though the phenomenon has been…
Towards Understanding the Word Sensitivity of Attention Layers: A Study via Random Features
Simone Bombari, Marco Mondelli
Understanding the reasons behind the exceptional success of transformers requires a better analysis of why attention layers are suitable for NLP tasks. In particular, such tasks re…
Deep Neural Collapse Is Provably Optimal for the Deep Unconstrained Features Model
Peter Súkeník, Marco Mondelli, Christoph Lampert
Neural collapse (NC) refers to the surprising structure of the last layer of deep neural networks in the terminal phase of gradient descent training. Recently, an increasing amount…
Improved Convergence of Score-Based Diffusion Models via Prediction-Correction
Francesco Pedrotti, Jan Maas, Marco Mondelli
Score-based generative models (SGMs) are powerful tools to sample from complex data distributions. Their underlying idea is to (i) run a forward process for time by adding no…
How Spurious Features Are Memorized: Precise Analysis for Random and NTK Features
Simone Bombari, Marco Mondelli
Deep learning models are known to overfit and memorize spurious features in the training dataset. While numerous empirical studies have aimed at understanding this phenomenon, a ri…
Concentration without Independence via Information Measures
Amedeo Roberto Esposito, Marco Mondelli
We propose a novel approach to concentration for non-independent random variables. The main idea is to ``pretend'' that the random variables are independent and pay a multiplicativ…
Mismatched estimation of non-symmetric rank-one matrices corrupted by structured noise
Teng Fu, YuHao Liu, Jean Barbier +3
We study the performance of a Bayesian statistician who estimates a rank-one signal corrupted by non-symmetric rotationally invariant noise with a generic distribution of singular…
Beyond the Universal Law of Robustness: Sharper Laws for Random Features and Neural Tangent Kernels
Simone Bombari, Shayan Kiyani, Marco Mondelli
Machine learning models are vulnerable to adversarial perturbations, and a thought-provoking paper by Bubeck and Sellke has analyzed this phenomenon through the lens of over-parame…
Fundamental Limits of Two-layer Autoencoders, and Achieving Them with Gradient Methods
Alexander Shevchenko, Kevin Kögler, Hamed Hassani +1
Autoencoders are a popular model in many branches of machine learning and lossy data compression. However, their fundamental limits, the performance of gradient methods and the fea…
Approximate Message Passing for Multi-Layer Estimation in Rotationally Invariant Models
Yizhou Xu, TianQi Hou, ShanSuo Liang +1
We consider the problem of reconstructing the signal and the hidden variables from observations coming from a multi-layer network with rotationally invariant weight matrices. The m…
Finite Sample Identification of Wide Shallow Neural Networks with Biases
Massimo Fornasier, Timo Klock, Marco Mondelli +1
Artificial neural networks are functions depending on a finite number of parameters typically encoded as weights and biases. The identification of the parameters of the network fro…
Precise Asymptotics for Spectral Methods in Mixed Generalized Linear Models
Yihan Zhang, Marco Mondelli, Ramji Venkataramanan
In a mixed generalized linear model, the goal is to learn multiple signals from unlabeled observations: each sample comes from exactly one signal, but it is not known which one. We…
Mean-field analysis for heavy ball methods: Dropout-stability, connectivity, and global convergence
Diyuan Wu, Vyacheslav Kungurtsev, Marco Mondelli
The stochastic heavy ball method (SHB), also known as stochastic gradient descent (SGD) with Polyak's momentum, is widely used in training neural networks. However, despite the rem…
Bayes-optimal limits in structured PCA, and how to reach them
Jean Barbier, Francesco Camilli, Marco Mondelli +1
How do statistical dependencies in measurement noise influence high-dimensional inference? To answer this, we study the paradigmatic spiked matrix model of principal components ana…
The price of ignorance: how much does it cost to forget noise structure in low-rank matrix estimation?
Jean Barbier, TianQi Hou, Marco Mondelli +1
We consider the problem of estimating a rank-1 signal corrupted by structured rotationally invariant noise, and address the following question: how well do inference algorithms per…
Memorization and Optimization in Deep Neural Networks with Minimum Over-parameterization
Simone Bombari, Mohammad Hossein Amani, Marco Mondelli
The Neural Tangent Kernel (NTK) has emerged as a powerful tool to provide memorization, optimization and generalization guarantees in deep neural networks. A line of work has studi…
Sharp asymptotics on the compression of two-layer neural networks
Mohammad Hossein Amani, Simone Bombari, Marco Mondelli +2
In this paper, we study the compression of a target two-layer neural network with N nodes into a compressed network with M<N nodes. More precisely, we consider the setting in which…
Polar Coded Computing: The Role of the Scaling Exponent
Dorsa Fathollahi, Marco Mondelli
We consider the problem of coded distributed computing using polar codes. The average execution time of a coded computing system is related to the error probability for transmissio…
Estimation in Rotationally Invariant Generalized Linear Models via Approximate Message Passing
Ramji Venkataramanan, Kevin Kögler, Marco Mondelli
We consider the problem of signal estimation in generalized linear models defined via rotationally invariant design matrices. Since these matrices can have an arbitrary spectral di…
Successive Syndrome-Check Decoding of Polar Codes
Seyyed Ali Hashemi, Marco Mondelli, John Cioffi +1
A two-part successive syndrome-check decoding of polar codes is proposed with the first part successively refining the received codeword and the second part checking its syndrome.…
Mean-field Analysis of Piecewise Linear Solutions for Wide ReLU Networks
Alexander Shevchenko, Vyacheslav Kungurtsev, Marco Mondelli
Understanding the properties of neural networks trained via stochastic gradient descent (SGD) is at the heart of the theory of deep learning. In this work, we take a mean-field vie…
Decoding Reed-Muller Codes with Successive Codeword Permutations
Nghia Doan, Seyyed Ali Hashemi, Marco Mondelli +1
A novel recursive list decoding (RLD) algorithm for Reed-Muller (RM) codes based on successive permutations (SP) of the codeword is presented. A low-complexity SP scheme applied to…
PCA Initialization for Approximate Message Passing in Rotationally Invariant Models
Marco Mondelli, Ramji Venkataramanan
We study the problem of estimating a rank- signal in the presence of rotationally invariant noise-a class of perturbations more general than Gaussian noise. Principal Component…