The geometry of nonlinear least squares with applications to sloppy models and optimization

Abstract

Parameter estimation by nonlinear least squares minimization is a common problem with an elegant geometric interpretation: the possible parameter values of a model induce a manifold in the space of data predictions. The minimization problem is then to find the point on the manifold closest to the data. We show that the model manifolds of a large class of models, known as sloppy models, have many universal features; they are characterized by a geometric series of widths, extrinsic curvatures, and parameter-effects curvatures. A number of common difficulties in optimizing least squares problems are due to this common structure. First, algorithms tend to run into the boundaries of the model manifold, causing parameters to diverge or become unphysical. We introduce the model graph as an extension of the model manifold to remedy this problem. We argue that appropriate priors can remove the boundaries and improve convergence rates. We show that typical fits will have many evaporated parameters. Second, bare model parameters are usually ill-suited to describing model behavior; cost contours in parameter space tend to form hierarchies of plateaus and canyons. Geometrically, we understand this inconvenient parametrization as an extremely skewed coordinate basis and show that it induces a large parameter-effects curvature on the manifold. Using coordinates based on geodesic motion, these narrow canyons are transformed in many cases into a single quadratic, isotropic basin. We interpret the modified Gauss-Newton and Levenberg-Marquardt fitting algorithms as an Euler approximation to geodesic motion in these natural coordinates on the model manifold and the model graph respectively. By adding a geodesic acceleration adjustment to these algorithms, we alleviate the difficulties from parameter-effects curvature, improving both efficiency and success rates at finding good fits.

40 pages, 29 Figures

References in corpus (5)

• Universally Sloppy Parameter Sensitivities in Systems Biology
• Sloppiness, robustness, and evolvability in systems biology
• Why are nonlinear fits so challenging?
• The sloppy model universality class and the Vandermonde matrix
• Extracting falsifiable predictions from sloppy models

Cited by in corpus (51)

• Parameter Space Compression Underlies Emergent Theories and Predictive Models
• The limitations of model-based experimental design and parameter estimation in sloppy systems
• Dihedral-angle-corrected registry-dependent interlayer potential for multilayer graphene structures
• PCA meets RG
• A critical review of statistical calibration/prediction models handling data inconsistency and model inadequacy
• Turning Statistical Physics Models Into Materials Design Engines
• Bridging Mechanistic and Phenomenological Models of Complex Biological Systems
• Maximizing the information learned from finite data selects a simple model
• Manifold learning for parameter reduction
• Delineating Parameter Unidentifiabilities in Complex Models
• The parameters uncertainty inflation fallacy
• Cause and Cure of Sloppiness in Ordinary Differential Equation Models
• Avalanche Spatial Structure and Multivariable Scaling Functions; Sizes, Heights, Widths, and Views through Windows
• Light Microscopy at Maximal Precision
• Sloppy nuclear energy density functionals: effective model reduction
• KLIFF: A framework to develop physics-based and machine learning interatomic potentials
• Bayesian, frequentist, and information geometric approaches to parametric uncertainty quantification of classical empirical interatomic potentials
• Chebyshev approximation and the global geometry of sloppy models
• Inferring the spatiotemporal DNA replication program from noisy data
• Strategic model reduction by analysing model sloppiness: a case study in coral calcification
• A KIM-compliant potfit for fitting sloppy interatomic potentials: Application to the EDIP model for silicon
• "Sloppy" nuclear energy density functionals (II): Finite nuclei
• Visualizing probabilistic models in Minkowski space with intensive symmetrized Kullback-Leibler embedding
• Structural Susceptibility and Separation of Time Scales in the van der Pol Oscillator
• Causal Geometry
• Signal detection in nearly continuous spectra and symmetry breaking
• The Training Process of Many Deep Networks Explores the Same Low-Dimensional Manifold
• Unwinding the model manifold: choosing similarity measures to remove local minima in sloppy dynamical systems
• Principles underlying efficient exciton transport unveiled by information-geometric analysis
• A regularization method for the parameter estimation problem in ordinary differential equations via discrete optimal control theory
• Accelerating Natural Gradient with Higher-Order Invariance
• Distinguishing Cell Phenotype Using Cell Epigenotype
• Dimension reduction for systems with slow relaxation
• Quantum Natural Gradient with Geodesic Corrections for Small Shallow Quantum Circuits
• Beyond Fisher Forecasting for Cosmology
• Does the Data Induce Capacity Control in Deep Learning?
• Stability of the manifold boundary approximation method for reductions of nuclear structure models
• Discovering sparse control strategies in C. elegans
• Sparse model from optimal nonuniform embedding of time series
• Analysis of sloppiness in model simulations: unveiling parameter uncertainty when mathematical models are fitted to data
• Applications of information geometry to spiking neural network behavior
• An information-matching approach to optimal experimental design and active learning
• Classical and Bayesian error analysis of the relativistic mean-field model for doubly magic nuclei
• Simmering: Sufficient is better than optimal for training neural networks
• Parameters inference and model reduction for the Single-Particle Model of Li ion cells
• Quantum Neural Network Training of a Repeater Node
• Information Geometry and the Renormalization Group
• Exploration of the Parameter Space in Macroeconomic Agent-Based Models
• Numerical parameter space compression and its application to microtubule dynamic instability
• A Monte Carlo approach to the conformal bootstrap
• Information loss under coarse-graining: a geometric approach