Local Saddle Point Optimization: A Curvature Exploitation Approach
arXiv:1805.05751
Abstract
Gradient-based optimization methods are the most popular choice for finding local optima for classical minimization and saddle point problems. Here, we highlight a systemic issue of gradient dynamics that arise for saddle point problems, namely the presence of undesired stable stationary points that are no local optima. We propose a novel optimization approach that exploits curvature information in order to escape from these undesired stationary points. We prove that different optimization methods, including gradient method and Adagrad, equipped with curvature exploitation can escape non-optimal stationary points. We also provide empirical results on common saddle point problems which confirm the advantage of using curvature exploitation.
References in corpus (6)
- Identifying and attacking the saddle point problem in high-dimensional non-convex optimization
- Certifying Some Distributional Robustness with Principled Adversarial Training
- A Variational Inequality Perspective on Generative Adversarial Networks
- Gradient Descent Converges to Minimizers
- Eigenvalues of the Hessian in Deep Learning: Singularity and Beyond
- Neon2: Finding Local Minima via First-Order Oracles
Cited by in corpus (9)
- Non-convex Min-Max Optimization: Applications, Challenges, and Recent Theoretical Advances
- What is Local Optimality in Nonconvex-Nonconcave Minimax Optimization?
- High-Resolution Mammogram Synthesis using Progressive Generative Adversarial Networks
- A Closer Look at the Optimization Landscapes of Generative Adversarial Networks
- Saddle Point Optimization with Approximate Minimization Oracle and its Application to Robust Berthing Control
- Negative Momentum for Improved Game Dynamics
- A Decentralized Proximal Point-type Method for Saddle Point Problems
- Saddle Point Optimization with Approximate Minimization Oracle
- Local saddle structure in relaxed averaged alternating reflections Algorithms on phase retrieval