Simmering: Sufficient is better than optimal for training neural networks

Abstract

The broad range of neural network training techniques that invoke optimization but rely on ad hoc modification for validity suggests that optimization-based training is misguided. Shortcomings of optimization-based training are brought to particularly strong relief by the problem of overfitting, where naive optimization produces spurious outcomes. The broad success of neural networks for modelling physical processes has prompted advances that are based on inverting the direction of investigation and treating neural networks as if they were physical systems in their own right. These successes raise the question of whether broader, physical perspectives could motivate the construction of improved training algorithms. Here, we introduce simmering, a physics-based method that trains neural networks to generate weights and biases that are merely ``good enough'', but which, paradoxically, outperforms leading optimization-based approaches. Using classification and regression examples we show that simmering corrects neural networks that are overfit by Adam, and show that simmering avoids overfitting if deployed from the outset. Our results question optimization as a paradigm for neural network training, and leverage information-geometric arguments to point to the existence of classes of sufficient training algorithms that do not take optimization as their starting point.

Minor corrections, clarifications

References in corpus (10)

• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
• An overview of gradient descent optimization algorithms
• Reconciling modern machine learning practice and the bias-variance trade-off
• The geometry of nonlinear least squares with applications to sloppy models and optimization
• On the approximation by single hidden layer feedforward neural networks with fixed weights
• Maximizing the information learned from finite data selects a simple model
• Visualizing probabilistic models: Intensive Principal Component Analysis
• Analysis of dropout learning regarded as ensemble learning
• Manifold boundaries give "gray-box" approximations of complex models
• Transforming Design Spaces Using Pareto-Laplace Filters