79 citations · 180 across the 19 of their papers we have counts for
3 papers · 1 filter
Rethinking the Variational Interpretation of Nesterov's Accelerated Method
Peiyuan Zhang, Antonio Orvieto, Hadi Daneshmand
The continuous-time model of Nesterov's momentum provides a thought-provoking perspective for understanding the nature of the acceleration phenomenon in convex optimization. One of…
Batch Normalization Orthogonalizes Representations in Deep Random Networks
Hadi Daneshmand, Amir Joudaki, Francis Bach
This paper underlines a subtle property of batch-normalization (BN): Successive batch normalizations with random linear transformations make hidden representations increasingly ort…
Revisiting the Role of Euler Numerical Integration on Acceleration and Stability in Convex Optimization
Peiyuan Zhang, Antonio Orvieto, Hadi Daneshmand +2
Viewing optimization methods as numerical integrators for ordinary differential equations (ODEs) provides a thought-provoking modern framework for studying accelerated first-order…