9 citations · 9 across the 6 of their papers we have counts for
6 papers · 1 filter
Towards Training Without Depth Limits: Batch Normalization Without Gradient Explosion
Alexandru Meterez, Amir Joudaki, Francesco Orabona +3
Normalization layers are one of the key building blocks for deep neural networks. Several theoretical studies have shown that batch normalization improves the signal propagation, b…
Local Saddle Point Optimization: A Curvature Exploitation Approach
Leonard Adolphs, Hadi Daneshmand, Aurelien Lucchi +1
Gradient-based optimization methods are the most popular choice for finding local optima for classical minimization and saddle point problems. Here, we highlight a systemic issue o…
Escaping Saddles with Stochastic Gradients
Hadi Daneshmand, Jonas Kohler, Aurelien Lucchi +1
We analyze the variance of stochastic gradients along negative curvature directions in certain non-convex machine learning models and show that stochastic gradients exhibit a stron…
Accelerated Dual Learning by Homotopic Initialization
Hadi Daneshmand, Hamed Hassani, Thomas Hofmann
Gradient descent and coordinate descent are well understood in terms of their asymptotic behavior, but less so in a transient regime often used for approximations in machine learni…
Adaptive Newton Method for Empirical Risk Minimization to Statistical Accuracy
Aryan Mokhtari, Alejandro Ribeiro
We consider empirical risk minimization for large-scale datasets. We introduce Ada Newton as an adaptive algorithm that uses Newton's method with adaptive sample sizes. The main id…
DynaNewton - Accelerating Newton's Method for Machine Learning
Hadi Daneshmand, Aurelien Lucchi, Thomas Hofmann
Newton's method is a fundamental technique in optimization with quadratic convergence within a neighborhood around the optimum. However reaching this neighborhood is often slow and…