Extreme Memorization via Scale of Initialization

Abstract

We construct an experimental setup in which changing the scale of initialization strongly impacts the implicit regularization induced by SGD, interpolating from good generalization performance to completely memorizing the training set while making little progress on the test set. Moreover, we find that the extent and manner in which generalization ability is affected depends on the activation and loss function used, with $\sin$ activation demonstrating extreme memorization. In the case of the homogeneous ReLU activation, we show that this behavior can be attributed to the loss function. Our empirical investigation reveals that increasing the scale of initialization correlates with misalignment of representations and gradients across examples in the same class. This insight allows us to devise an alignment measure over gradients and representations which can capture this phenomenon. We demonstrate that our alignment measure correlates with generalization of deep models trained on image classification tasks.

References in corpus (14)

• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• A Convergence Theory for Deep Learning via Over-Parameterization
• Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data
• Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data
• Path-SGD: Path-Normalized Optimization in Deep Neural Networks
• Fisher-Rao Metric, Geometry, and Complexity of Neural Networks
• To understand deep learning we need to understand kernel learning
• The Shattered Gradients Problem: If resnets are the answer, then what is the question?
• Norm-Based Capacity Control in Neural Networks
• Generalization in Deep Networks: The Role of Distance from Initialization
• Kernel and Rich Regimes in Overparametrized Models
• Inductive Bias of Deep Convolutional Networks through Pooling Geometry
• The Impact of Neural Network Overparameterization on Gradient Confusion and Stochastic Gradient Descent
• Coherent Gradients: An Approach to Understanding Generalization in Gradient Descent-based Optimization

Cited by in corpus (1)

• Stochastic Training is Not Necessary for Generalization