70 citations · 85 across the 19 of their papers we have counts for
9 papers · 1 filter
Exact Dynamics of Multi-class Stochastic Gradient Descent
Elizabeth Collins-Woodfin, Inbar Seroussi
We develop a framework for analyzing the learning dynamics of high-dimensional problems trained using one-pass stochastic gradient descent (SGD) with data from multiple anisotropic…
Minimax Rates for Learning Pairwise Interactions in Attention-Style Models
Shai Zucker, Xiong Wang, Fei Lu +1
We study the convergence rate of learning pairwise interactions in single-layer attention-style models, where tokens interact through a weight matrix and a nonlinear activation fun…
High-Dimensional Private Linear Regression with Optimal Rates
Simone Bombari, Jialei Luo, Inbar Seroussi +1
Differentially private (DP) linear regression has received significant attention in the recent theoretical literature, with several approaches proposed to improve error rates. Our…
Applications of Statistical Field Theory in Deep Learning
Zohar Ringel, Noa Rubin, Edo Mor +2
Deep learning algorithms have made incredible strides in the past decade, yet due to their complexity, the science of deep learning remains in its early stages. Being an experiment…
Grokking as a First Order Phase Transition in Two Layer Networks
Noa Rubin, Inbar Seroussi, Zohar Ringel
A key property of deep neural networks (DNNs) is their ability to learn new features during training. This intriguing aspect of deep learning stands out most clearly in recently re…
Speed Limits for Deep Learning
Inbar Seroussi, Alexander A. Alemi, Moritz Helias +1
State-of-the-art neural networks require extreme computational power to train. It is therefore natural to wonder whether they are optimally trained. Here we apply a recent advancem…
Spectral-Bias and Kernel-Task Alignment in Physically Informed Neural Networks
Inbar Seroussi, Asaf Miron, Zohar Ringel
Physically informed neural networks (PINNs) are a promising emerging method for solving differential equations. As in many other deep learning approaches, the choice of PINN design…
Separation of Scales and a Thermodynamic Description of Feature Learning in Some CNNs
Inbar Seroussi, Gadi Naveh, Zohar Ringel
Deep neural networks (DNNs) are powerful tools for compressing and distilling information. Their scale and complexity, often involving billions of inter-dependent parameters, rende…
Lower Bounds on the Generalization Error of Nonlinear Learning Models
Inbar Seroussi, Ofer Zeitouni
We study in this paper lower bounds for the generalization error of models derived from multi-layer neural networks, in the regime where the size of the layers is commensurate with…