Traversing the noise of dynamic mini-batch sub-sampled loss functions: A visual guide
arXiv:1903.08552
Abstract
Mini-batch sub-sampling in neural network training is unavoidable, due to growing data demands, memory-limited computational resources such as graphical processing units (GPUs), and the dynamics of on-line learning. In this study we specifically distinguish between static mini-batch sub-sampled loss functions, where mini-batches are intermittently fixed during training, resulting in smooth but biased loss functions; and the dynamic sub-sampling equivalent, where new mini-batches are sampled at every loss evaluation, trading bias for variance in sampling induced discontinuities. These render automated optimization strategies such as minimization line searches ineffective, since critical points may not exist and function minimizers find spurious, discontinuity induced minima. This paper suggests recasting the optimization problem to find stochastic non-negative associated gradient projection points (SNN-GPPs). We demonstrate that the SNN-GPP optimality criterion is less susceptible to sub-sampling induced discontinuities than critical points or minimizers. We conduct a visual investigation, comparing local minimum and SNN-GPP optimality criteria in the loss functions of a simple neural network training problem for a variety of popular activation functions. Since SNN-GPPs better approximate the location of true optima, particularly when using smooth activation functions with high curvature characteristics, we postulate that line searches locating SNN-GPPs can contribute significantly to automating neural network training
arXiv comment43 pages, 22 Figures, to be submitted to a journal
References in corpus (14)
- The Loss Surfaces of Multilayer Networks
- Visualizing the Loss Landscape of Neural Nets
- Identifying and attacking the saddle point problem in high-dimensional non-convex optimization
- Revisiting Small Batch Training for Deep Neural Networks
- Qualitatively characterizing neural network optimization problems
- On the loss landscape of a class of deep neural networks with no bad local valleys
- Revise Saturated Activation Functions
- Adding One Neuron Can Eliminate All Bad Local Minima
- An empirical analysis of the optimization of deep network loss surfaces
- A Stochastic Line Search Method with Convergence Rate Analysis
- A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networks
- Reinforcement Learning for Learning Rate Control
- Gradient-only line searches: An Alternative to Probabilistic Line Searches
- Algorithms for solving optimization problems arising from deep neural net models: smooth problems