Convergence of adaptive algorithms for weakly convex constrained optimization

Abstract

We analyze the adaptive first order algorithm AMSGrad, for solving a constrained stochastic optimization problem with a weakly convex objective. We prove the $\tilde{\mathcal{O}}(t^{-1/4})$ rate of convergence for the norm of the gradient of Moreau envelope, which is the standard stationarity measure for this class of problems. It matches the known rates that adaptive algorithms enjoy for the specific case of unconstrained smooth stochastic optimization. Our analysis works with mini-batch size of $1$, constant first and second order moment parameters, and possibly unbounded optimization domains. Finally, we illustrate the applications and extensions of our results to specific problems and algorithms.

References in corpus (8)

• On the Convergence of Adam and Beyond
• Black-box Adversarial Attacks with Limited Queries and Information
• AdaGrad stepsizes: Sharp convergence over nonconvex landscapes
• Imposing Hard Constraints on Deep Networks: Promises and Limitations
• ZO-AdaMM: Zeroth-Order Adaptive Momentum Method for Black-Box Optimization
• On the Convergence of Stochastic Gradient Descent with Adaptive Stepsizes
• On Connections between Constrained Optimization and Reinforcement Learning
• Convergence Analysis of a Momentum Algorithm with Adaptive Step Size for Non Convex Optimization

Cited by in corpus (1)

• Stability and Convergence of Stochastic Gradient Clipping: Beyond Lipschitz Continuity and Smoothness