NCVX: A User-Friendly and Scalable Package for Nonconvex Optimization in Machine Learning
arXiv:2111.13984
Abstract
Optimizing nonconvex (NCVX) problems, especially nonsmooth and constrained ones, is an essential part of machine learning. However, it can be hard to reliably solve such problems without optimization expertise. Existing general-purpose NCVX optimization packages are powerful but typically cannot handle nonsmoothness. GRANSO is among the first optimization solvers targeting general nonsmooth NCVX problems with nonsmooth constraints, but, as it is implemented in MATLAB and requires the user to provide analytical gradients, GRANSO is often not a convenient choice in machine learning (especially deep learning) applications. To greatly lower the technical barrier, we introduce a new software package called NCVX, whose initial release contains the solver PyGRANSO, a PyTorch-enabled port of GRANSO incorporating auto-differentiation, GPU acceleration, tensor input, and support for new QP solvers. NCVX is built on freely available and widely used open-source frameworks, and as a highlight, can solve general constrained deep learning problems, the first of its kind. NCVX is available at https://ncvx.org, with detailed documentation and numerous examples from machine learning and other fields.
arXiv commentNCVX is available at https://ncvx.org
References in corpus (12)
- Scikit-learn: Machine Learning in Python
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- OSQP: An Operator Splitting Solver for Quadratic Programs
- Manopt, a Matlab toolbox for optimization on manifolds
- Templates for Convex Cone Problems with Applications to Sparse Signal Recovery
- Identifying and attacking the saddle point problem in high-dimensional non-convex optimization
- Optimization for deep learning: theory and algorithms
- What is Local Optimality in Nonconvex-Nonconcave Minimax Optimization?
- Perceptual Adversarial Robustness: Defense Against Unseen Threat Models
- Conic Optimization Theory: Convexification Techniques and Numerical Algorithms
- GENO -- GENeric Optimization for Classical Machine Learning
- The ensmallen library for flexible numerical optimization