1.9k citations · 6.5k across the 132 of their papers we have counts for
137 papers
Research Program: Theory of Learning in Dynamical Systems
Elad Hazan, Shai Shalev Shwartz, Nathan Srebro
Modern learning systems increasingly interact with data that evolve over time and depend on hidden internal state. We ask a basic question: when is such a dynamical system learnabl…
From Continual Learning to SGD and Back: Better Rates for Continual Linear Models
Itay Evron, Ran Levinstein, Matan Schliserman +4
We study the common continual learning setup where an overparameterized model is sequentially fitted to a set of jointly realizable tasks. We analyze forgetting, defined as the los…
Provable Tempered Overfitting of Minimal Nets and Typical Nets
Itamar Harel, William M. Hoza, Gal Vardi +3
We study the overfitting behavior of fully connected deep Neural Networks (NNs) with binary weights fitted to perfectly classify a noisy training set. We consider interpolation usi…
Overfitting Behaviour of Gaussian Kernel Ridgeless Regression: Varying Bandwidth or Dimensionality
Marko Medvedev, Gal Vardi, Nathan Srebro
We consider the overfitting behavior of minimum norm interpolating solutions of Gaussian kernel ridge regression (i.e. kernel ridgeless regression), when the bandwidth or input dim…
The Limits and Potentials of Local SGD for Distributed Heterogeneous Learning with Intermittent Communication
Kumar Kshitij Patel, Margalit Glasgow, Ali Zindari +5
Local SGD is a popular optimization method in distributed learning, often outperforming other algorithms in practice, including mini-batch SGD. Despite this success, theoretically…
Depth Separation in Norm-Bounded Infinite-Width Neural Networks
Suzanna Parkinson, Greg Ongie, Rebecca Willett +2
We study depth separation in infinite-width neural networks, where complexity is controlled by the overall squared -norm of the weights (sum of squares of all weights in th…