What Do Neural Networks Learn When Trained With Random Labels?

Abstract

We study deep neural networks (DNNs) trained on natural image data with entirely random labels. Despite its popularity in the literature, where it is often used to study memorization, generalization, and other phenomena, little is known about what DNNs learn in this setting. In this paper, we show analytically for convolutional and fully connected networks that an alignment between the principal components of network parameters and data takes place when training with random labels. We study this alignment effect by investigating neural networks pre-trained on randomly labelled image data and subsequently fine-tuned on disjoint datasets with random or real labels. We show how this alignment produces a positive transfer: networks pre-trained with random labels train faster downstream compared to training from scratch even after accounting for simple effects, such as weight scaling. We analyze how competing effects, such as specialization at later layers, may hide the positive transfer. These effects are studied in several network architectures, including VGG16 and ResNet18, on CIFAR10 and ImageNet.

arXiv commentAccepted, NeurIPS2020

References in corpus (19)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• How transferable are features in deep neural networks?
• Language Models are Few-Shot Learners
• Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)
• Classification with Noisy Labels by Importance Reweighting
• Training Convolutional Networks with Noisy Labels
• A Closer Look at Memorization in Deep Networks
• Sensitivity and Generalization in Neural Networks: an Empirical Study
• Dimensionality-Driven Learning with Noisy Labels
• Intrinsic dimension of data representations in deep neural networks
• Norm-Based Capacity Control in Neural Networks
• The Early Phase of Neural Network Training
• A PCA-Based Convolutional Network
• Network Deconvolution
• DNN or k-NN: That is the Generalize vs. Memorize Question
• Neural Network Memorization Dissection
• Leveraging Random Label Memorization for Unsupervised Pre-Training
• Detecting Memorization in ReLU Networks

Cited by in corpus (11)

• FINE Samples for Learning with Noisy Labels
• A Contrastive Variational Graph Auto-Encoder for Node Clustering
• Gaussian Universality of Perceptrons with Random Labels
• ReFine: Re-randomization before Fine-tuning for Cross-domain Few-shot Learning
• The Impact of Reinitialization on Generalization in Convolutional Neural Networks
• Neural Architecture Search with Random Labels
• Slot Machines: Discovering Winning Combinations of Random Weights in Neural Networks
• Energy Attack: On Transferring Adversarial Examples
• Compositional generalization in semantic parsing with pretrained transformers
• Does Pretraining for Summarization Require Knowledge Transfer?
• Regularization by Misclassification in ReLU Neural Networks