2.2k citations · 6.4k across the 57 of their papers we have counts for
14 papers · 1 filter
Task Agnostic Continual Learning Using Online Variational Bayes with Fixed-Point Updates
Chen Zeno, Itay Golan, Elad Hoffer +1
Background: Catastrophic forgetting is the notorious vulnerability of neural networks to the changes in the data distribution during learning. This phenomenon has long been conside…
A Mean Field Theory of Quantized Deep Networks: The Quantization-Depth Trade-Off
Yaniv Blumenfeld, Dar Gilboa, Daniel Soudry
Reducing the precision of weights and activation functions in neural network training, with minimal impact on performance, is essential for the deployment of these models in resour…
Lexicographic and Depth-Sensitive Margins in Homogeneous and Non-Homogeneous Deep Models
Mor Shpigel Nacson, Suriya Gunasekar, Jason D. Lee +2
With an eye toward understanding complexity control in deep learning, we study how infinitesimal regularization or gradient descent optimization lead to margin maximizing solutions…
Stochastic Gradient Descent on Separable Data: Exact Convergence with a Fixed Learning Rate
Mor Shpigel Nacson, Nathan Srebro, Daniel Soudry
Stochastic Gradient Descent (SGD) is a central tool in machine learning. We prove that SGD converges to zero loss, even with a fixed (non-vanishing) learning rate - in the special…
Task Agnostic Continual Learning Using Online Variational Bayes
Chen Zeno, Itay Golan, Elad Hoffer +1
Catastrophic forgetting is the notorious vulnerability of neural networks to the change of the data distribution while learning. This phenomenon has long been considered a major ob…
Convergence of Gradient Descent on Separable Data
Mor Shpigel Nacson, Jason D. Lee, Suriya Gunasekar +3
We provide a detailed study on the implicit bias of gradient descent when optimizing loss functions with strictly monotone tails, such as the logistic loss, over separable datasets…