LCA: Loss Change Allocation for Neural Network Training

Abstract

Neural networks enjoy widespread use, but many aspects of their training, representation, and operation are poorly understood. In particular, our view into the training process is limited, with a single scalar loss being the most common viewport into this high-dimensional, dynamic process. We propose a new window into training called Loss Change Allocation (LCA), in which credit for changes to the network loss is conservatively partitioned to the parameters. This measurement is accomplished by decomposing the components of an approximate path integral along the training trajectory using a Runge-Kutta integrator. This rich view shows which parameters are responsible for decreasing or increasing the loss during training, or which parameters "help" or "hurt" the network's learning, respectively. LCA may be summed over training iterations and/or over neurons, channels, or layers for increasingly coarse views. This new measurement device produces several insights into training. (1) We find that barely over 50% of parameters help during any given iteration. (2) Some entire layers hurt overall, moving on average against the training gradient, a phenomenon we hypothesize may be due to phase lag in an oscillatory training process. (3) Finally, increments in learning proceed in a synchronized manner across layers, often peaking on identical iterations.

arXiv commentNeurIPS 2019 camera ready version

References in corpus (16)

• Overcoming catastrophic forgetting in neural networks
• Improving neural networks by preventing co-adaptation of feature detectors
• Understanding Neural Networks Through Deep Visualization
• The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
• Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
• Opening the Black Box of Deep Neural Networks via Information
• The Loss Surfaces of Multilayer Networks
• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
• Qualitatively characterizing neural network optimization problems
• The loss surface of deep and wide neural networks
• No bad local minima: Data independent training error guarantees for multilayer neural networks
• Are All Layers Created Equal?
• Deconstructing Lottery Tickets: Zeros, Signs, and the Supermask
• Critical Learning Periods in Deep Neural Networks
• A Walk with SGD
• On the Relation Between the Sharpest Directions of DNN Loss and the SGD Step Length

Cited by in corpus (6)

• SpaceNet: Make Free Space For Continual Learning
• Exploring the Vulnerability of Deep Neural Networks: A Study of Parameter Corruption
• Adversarial Parameter Defense by Multi-Step Risk Minimization
• Unifying Regularisation Methods for Continual Learning
• TRADI: Tracking deep neural network weight distributions for uncertainty estimation
• Convolutional neural networks compression with low rank and sparse tensor decompositions