70 citations · 209 across the 45 of their papers we have counts for
12 papers · 1 filter
An Adaptive Stochastic Gradient Method with Non-negative Gauss-Newton Stepsizes
Antonio Orvieto, Lin Xiao
We consider the problem of minimizing the average of a large number of smooth but possibly non-convex functions. In the context of most machine learning applications, each loss fun…
An Accelerated Lyapunov Function for Polyak's Heavy-Ball on Convex Quadratics
Antonio Orvieto
In 1964, Polyak showed that the Heavy-ball method, the simplest momentum technique, accelerates convergence of strongly-convex problems in the vicinity of the solution. While Neste…
On the Theoretical Properties of Noise Correlation in Stochastic Optimization
Aurelien Lucchi, Frank Proske, Antonio Orvieto +2
Studying the properties of stochastic noise to optimize complex non-convex functions has been an active area of research in the field of machine learning. Prior work has shown that…
Dynamics of SGD with Stochastic Polyak Stepsizes: Truly Adaptive Variants and Convergence to Exact Solution
Antonio Orvieto, Simon Lacoste-Julien, Nicolas Loizou
Recently, Loizou et al. (2021), proposed and analyzed stochastic gradient descent (SGD) with stochastic Polyak stepsize (SPS). The proposed SPS comes with strong convergence guaran…
On the Second-order Convergence Properties of Random Search Methods
Aurelien Lucchi, Antonio Orvieto, Adamos Solomou
We study the theoretical convergence properties of random-search methods when optimizing non-convex objective functions without having access to derivatives. We prove that standard…
Rethinking the Variational Interpretation of Nesterov's Accelerated Method
Peiyuan Zhang, Antonio Orvieto, Hadi Daneshmand
The continuous-time model of Nesterov's momentum provides a thought-provoking perspective for understanding the nature of the acceleration phenomenon in convex optimization. One of…
Revisiting the Role of Euler Numerical Integration on Acceleration and Stability in Convex Optimization
Peiyuan Zhang, Antonio Orvieto, Hadi Daneshmand +2
Viewing optimization methods as numerical integrators for ordinary differential equations (ODEs) provides a thought-provoking modern framework for studying accelerated first-order…
An Accelerated DFO Algorithm for Finite-sum Convex Functions
Yuwen Chen, Antonio Orvieto, Aurelien Lucchi
Derivative-free optimization (DFO) has recently gained a lot of momentum in machine learning, spawning interest in the community to design faster methods for problems where gradien…
Momentum Improves Optimization on Riemannian Manifolds
Foivos Alimisis, Antonio Orvieto, Gary Bécigneul +1
We develop a new Riemannian descent algorithm that relies on momentum to improve over existing first-order methods for geodesically convex optimization. In contrast, accelerated co…
Shadowing Properties of Optimization Algorithms
Antonio Orvieto, Aurelien Lucchi
Ordinary differential equation (ODE) models of gradient-based optimization methods can provide insights into the dynamics of learning and inspire the design of new algorithms. Unfo…
A Continuous-time Perspective for Modeling Acceleration in Riemannian Optimization
Foivos Alimisis, Antonio Orvieto, Gary Bécigneul +1
We propose a novel second-order ODE as the continuous-time limit of a Riemannian accelerated gradient-based method on a manifold with curvature bounded from below. This ODE can be…
Continuous-time Models for Stochastic Optimization Algorithms
Antonio Orvieto, Aurelien Lucchi
We propose new continuous-time formulations for first-order stochastic optimization algorithms such as mini-batch gradient descent and variance-reduced methods. We exploit these co…