Whitening and second order optimization both make information in the dataset unusable during training, and can reduce or prevent generalization

Abstract

Machine learning is predicated on the concept of generalization: a model achieving low error on a sufficiently large training set should also perform well on novel samples from the same distribution. We show that both data whitening and second order optimization can harm or entirely prevent generalization. In general, model training harnesses information contained in the sample-sample second moment matrix of a dataset. For a general class of models, namely models with a fully connected first layer, we prove that the information contained in this matrix is the only information which can be used to generalize. Models trained using whitened data, or with certain second order optimization schemes, have less access to this information, resulting in reduced or nonexistent generalization ability. We experimentally verify these predictions for several architectures, and further demonstrate that generalization continues to be harmed even when theoretical requirements are relaxed. However, we also show experimentally that regularized second order optimization can provide a practical tradeoff, where training is accelerated but less information is lost, and generalization can in some circumstances even improve.

arXiv comment13+10 pages, 10 figures; minor textual changes and some reorganization, one new figure and a new proof of main theorem added

References in corpus (22)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• ADADELTA: An Adaptive Learning Rate Method
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Opening the Black Box of Deep Neural Networks via Information
• Optimizing Neural Networks with Kronecker-factored Approximate Curvature
• Detecting the stochastic gravitational wave background using pulsar timing
• Adafactor: Adaptive Learning Rates with Sublinear Memory Cost
• Measuring the Effects of Data Parallelism on Neural Network Training
• On the Spectral Bias of Neural Networks
• Krylov Subspace Descent for Deep Learning
• Natural Neural Networks
• Shampoo: Preconditioned Stochastic Tensor Optimization
• Three Mechanisms of Weight Decay Regularization
• Where is the Information in a Deep Neural Network?
• Learners that Use Little Information
• Dynamics of Deep Neural Networks and Neural Tangent Hierarchy
• Practical Gauss-Newton Optimisation for Deep Learning
• Block-diagonal Hessian-free Optimization for Training Neural Networks
• Information in Infinite Ensembles of Infinitely-Wide Neural Networks
• The Natural Gradient by Analogy to Signal Whitening, and Recipes and Tricks for its Use
• To Each Optimizer a Norm, To Each Norm its Generalization
• Block Mean Approximation for Efficient Second Order Optimization