26 citations · 137 across the 41 of their papers we have counts for
12 papers · 1 filter
Privacy for Free in the Overparameterized Regime
Simone Bombari, Marco Mondelli
Differentially private gradient descent (DP-GD) is a popular algorithm to train deep learning models with provable guarantees on the privacy of the training data. In the last decad…
Towards Understanding the Word Sensitivity of Attention Layers: A Study via Random Features
Simone Bombari, Marco Mondelli
Understanding the reasons behind the exceptional success of transformers requires a better analysis of why attention layers are suitable for NLP tasks. In particular, such tasks re…
How Spurious Features Are Memorized: Precise Analysis for Random and NTK Features
Simone Bombari, Marco Mondelli
Deep learning models are known to overfit and memorize spurious features in the training dataset. While numerous empirical studies have aimed at understanding this phenomenon, a ri…
Beyond the Universal Law of Robustness: Sharper Laws for Random Features and Neural Tangent Kernels
Simone Bombari, Shayan Kiyani, Marco Mondelli
Machine learning models are vulnerable to adversarial perturbations, and a thought-provoking paper by Bubeck and Sellke has analyzed this phenomenon through the lens of over-parame…
Approximate Message Passing for Multi-Layer Estimation in Rotationally Invariant Models
Yizhou Xu, TianQi Hou, ShanSuo Liang +1
We consider the problem of reconstructing the signal and the hidden variables from observations coming from a multi-layer network with rotationally invariant weight matrices. The m…
Memorization and Optimization in Deep Neural Networks with Minimum Over-parameterization
Simone Bombari, Mohammad Hossein Amani, Marco Mondelli
The Neural Tangent Kernel (NTK) has emerged as a powerful tool to provide memorization, optimization and generalization guarantees in deep neural networks. A line of work has studi…
Estimation in Rotationally Invariant Generalized Linear Models via Approximate Message Passing
Ramji Venkataramanan, Kevin Kögler, Marco Mondelli
We consider the problem of signal estimation in generalized linear models defined via rotationally invariant design matrices. Since these matrices can have an arbitrary spectral di…
PCA Initialization for Approximate Message Passing in Rotationally Invariant Models
Marco Mondelli, Ramji Venkataramanan
We study the problem of estimating a rank- signal in the presence of rotationally invariant noise-a class of perturbations more general than Gaussian noise. Principal Component…
Tight Bounds on the Smallest Eigenvalue of the Neural Tangent Kernel for Deep ReLU Networks
Quynh Nguyen, Marco Mondelli, Guido Montufar
A recent line of work has analyzed the theoretical properties of deep neural networks via the Neural Tangent Kernel (NTK). In particular, the smallest eigenvalue of the NTK has bee…
Approximate Message Passing with Spectral Initialization for Generalized Linear Models
Marco Mondelli, Ramji Venkataramanan
We consider the problem of estimating a signal from measurements obtained via a generalized linear model. We focus on estimators based on approximate message passing (AMP), a famil…
Optimal Combination of Linear and Spectral Estimators for Generalized Linear Models
Marco Mondelli, Christos Thrampoulidis, Ramji Venkataramanan
We study the problem of recovering an unknown signal given measurements obtained from a generalized linear model with a Gaussian sensing matrix. Two popular solutio…
Fundamental Limits of Weak Recovery with Applications to Phase Retrieval
Marco Mondelli, Andrea Montanari
In phase retrieval we want to recover an unknown signal from quadratic measurements of the form $y_i = |\langle{\boldsymbol a}_i,{\boldsymbol x}\r…