79 citations · 180 across the 19 of their papers we have counts for
3 papers · 1 filter
Exponential convergence rates for Batch Normalization: The power of length-direction decoupling in non-convex optimization
Jonas Kohler, Hadi Daneshmand, Aurelien Lucchi +3
Normalization techniques such as Batch Normalization have been applied successfully for training deep neural networks. Yet, despite its apparent empirical benefits, the reasons beh…
Local Saddle Point Optimization: A Curvature Exploitation Approach
Leonard Adolphs, Hadi Daneshmand, Aurelien Lucchi +1
Gradient-based optimization methods are the most popular choice for finding local optima for classical minimization and saddle point problems. Here, we highlight a systemic issue o…
Escaping Saddles with Stochastic Gradients
Hadi Daneshmand, Jonas Kohler, Aurelien Lucchi +1
We analyze the variance of stochastic gradients along negative curvature directions in certain non-convex machine learning models and show that stochastic gradients exhibit a stron…