A Closer Look at Memorization in Deep Networks

Abstract

We examine the role of memorization in deep learning, drawing connections to capacity, generalization, and adversarial robustness. While deep networks are capable of memorizing noise data, our results suggest that they tend to prioritize learning simple patterns first. In our experiments, we expose qualitative differences in gradient-based optimization of deep neural networks (DNNs) on noise vs. real data. We also demonstrate that for appropriately tuned explicit regularization (e.g., dropout) we can degrade DNN training performance on noise datasets without compromising generalization on real data. Our analysis suggests that the notions of effective capacity which are dataset independent are unlikely to explain the generalization performance of deep networks when trained with gradient based methods because training data itself plays an important role in determining the degree of memorization.

Appears in Proceedings of the 34th International Conference on Machine Learning (ICML 2017), Devansh Arpit, Stanisław Jastrzębski, Nicolas Ballas, and David Krueger contributed equally to this work

References in corpus (3)

• Explaining and Harnessing Adversarial Examples
• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
• Gradient-based Hyperparameter Optimization through Reversible Learning

Cited by in corpus (44)

• Deep Learning Scaling is Predictable, Empirically
• How does Disagreement Help Generalization against Label Corruption?
• Overfitting Mechanism and Avoidance in Deep Neural Networks
• The no-free-lunch theorems of supervised learning
• Convolutional neural network identification of galaxy post-mergers in UNIONS using IllustrisTNG
• Generalization in Deep Networks: The Role of Distance from Initialization
• Unsupervised Domain Adaptation of Black-Box Source Models
• Explicitizing an Implicit Bias of the Frequency Principle in Two-layer Neural Networks
• Flow-Mixup: Classifying Multi-labeled Medical Images with Corrupted Labels
• An LSTM approach to Forecast Migration using Google Trends
• Source-free Domain Adaptation via Avatar Prototype Generation and Adaptation
• Understanding the Interaction of Adversarial Training with Noisy Labels
• SemiNLL: A Framework of Noisy-Label Learning by Semi-Supervised Learning
• Consensual Collaborative Training And Knowledge Distillation Based Facial Expression Recognition Under Noisy Annotations
• Augmentation Strategies for Learning with Noisy Labels
• Combating Noise: Semi-supervised Learning by Region Uncertainty Quantification
• Affect Expression Behaviour Analysis in the Wild using Consensual Collaborative Training
• Meta-learners' learning dynamics are unlike learners'
• Instance Correction for Learning with Open-set Noisy Labels
• A Closer Look at Data Bias in Neural Extractive Summarization Models
• Providing theoretical learning guarantees to Deep Learning Networks
• Rethinking Generalization of Neural Models: A Named Entity Recognition Case Study
• On the Effectiveness of Regularization Against Membership Inference Attacks
• A Meta Approach to Defend Noisy Labels by the Manifold Regularizer PSDR
• Robust and On-the-fly Dataset Denoising for Image Classification
• A Unified Plug-and-Play Framework for Effective Data Denoising and Robust Abstention
• MetaInfoNet: Learning Task-Guided Information for Sample Reweighting
• Investigating performance of neural networks and gradient boosting models approximating microscopic traffic simulations in traffic optimization tasks
• Robust Deep Learning with Active Noise Cancellation for Spatial Computing
• A Simple yet Effective Baseline for Robust Deep Learning with Noisy Labels
• On the Orthogonality of Knowledge Distillation with Other Techniques: From an Ensemble Perspective
• Intraclass clustering: an implicit learning ability that regularizes DNNs
• Network Cooperation with Progressive Disambiguation for Partial Label Learning
• Impact of Training Dataset Size on Neural Answer Selection Models
• Phases of learning dynamics in artificial neural networks: with or without mislabeled data
• Memory and attention in deep learning
• ComSum: Commit Messages Summarization and Meaning Preservation
• Noisy-Labeled NER with Confidence Estimation
• Learning to Rectify for Robust Learning with Noisy Labels
• Learning the gravitational force law and other analytic functions
• An investigation of model-free planning
• Neural Networks and Denotation
• Batch Inverse-Variance Weighting: Deep Heteroscedastic Regression
• A Theoretical Analysis of Learning with Noisily Labeled Data