Conformal Symplectic and Relativistic Optimization

Abstract

Arguably, the two most popular accelerated or momentum-based optimization methods in machine learning are Nesterov's accelerated gradient and Polyaks's heavy ball, both corresponding to different discretizations of a particular second order differential equation with friction. Such connections with continuous-time dynamical systems have been instrumental in demystifying acceleration phenomena in optimization. Here we study structure-preserving discretizations for a certain class of dissipative (conformal) Hamiltonian systems, allowing us to analyze the symplectic structure of both Nesterov and heavy ball, besides providing several new insights into these methods. Moreover, we propose a new algorithm based on a dissipative relativistic system that normalizes the momentum and may result in more stable/faster optimization. Importantly, such a method generalizes both Nesterov and heavy ball, each being recovered as distinct limiting cases, and has potential advantages at no additional cost.

arXiv commentA short version of this paper appeared at NeurIPS 2020 (spotlight). This lengthier version matches the published paper at JSTAT, which contains additional results

References in corpus (6)

• A Literature Survey of Benchmark Functions For Global Optimization Problems
• A Variational Perspective on Accelerated Methods in Optimization
• ADMM and Accelerated ADMM as Continuous Dynamical Systems
• Aubry-Mather Theory for Conformally Symplectic Systems
• A Nonsmooth Dynamical Systems Perspective on Accelerated Extensions of ADMM
• On dissipative symplectic integration with applications to gradient-based optimization

Cited by in corpus (7)

• A Nonsmooth Dynamical Systems Perspective on Accelerated Extensions of ADMM
• On dissipative symplectic integration with applications to gradient-based optimization
• Gradient flows and proximal splitting methods: A unified view on accelerated and stochastic optimization
• Chaotic saddles and interior crises in a dissipative nontwist system
• Geometric Methods for Sampling, Optimisation, Inference and Adaptive Agents
• Conformal Symplectic Optimization for Stable Reinforcement Learning
• A geometric integration approach to smooth optimisation: Foundations of the discrete gradient method