Invexifying Regularization of Non-Linear Least-Squares Problems

Abstract

We consider regularization of non-convex optimization problems involving a non-linear least-squares objective. By adding an auxiliary set of variables, we introduce a novel regularization framework whose corresponding objective function is not only provably invex, but it also satisfies the highly desirable Polyak--Lojasiewicz inequality for any choice of the regularization parameter. Although our novel framework is entirely different from the classical ${\ell }_2$-regularization, an interesting connection is established for the special case of under-determined linear least-squares. In particular, we show that gradient descent applied to our novel regularized formulation converges to the same solution as the linear ridge-regression problem. Numerical experiments corroborate our theoretical results and demonstrate the method's performance in practical situations as compared to the typical ${\ell }_2$-regularization.

arXiv comment18 pages

References in corpus (19)

• Adam: A Method for Stochastic Optimization
• Decoupled Weight Decay Regularization
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Improving neural networks by preventing co-adaptation of feature detectors
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• A Conceptual Introduction to Hamiltonian Monte Carlo
• Demystifying MMD GANs
• PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimization
• On exponential convergence of SGD in non-convex over-parametrized learning
• Overparameterized Nonlinear Learning: Gradient Descent Takes the Shortest Path?
• Shampoo: Preconditioned Stochastic Tensor Optimization
• Three Mechanisms of Weight Decay Regularization
• Elimination of All Bad Local Minima in Deep Learning
• On the Computation and Communication Complexity of Parallel SGD with Dynamic Batch Sizes for Stochastic Non-Convex Optimization
• Inexact Non-Convex Newton-Type Methods
• Stochastic Variance-Reduced Cubic Regularization for Nonconvex Optimization
• On the Convergence of SGD with Biased Gradients
• Convergence of Newton-MR under Inexact Hessian Information
• Stagewise Training Accelerates Convergence of Testing Error Over SGD