Estimating informativeness of samples with Smooth Unique Information

Abstract

We define a notion of information that an individual sample provides to the training of a neural network, and we specialize it to measure both how much a sample informs the final weights and how much it informs the function computed by the weights. Though related, we show that these quantities have a qualitatively different behavior. We give efficient approximations of these quantities using a linearized network and demonstrate empirically that the approximation is accurate for real-world architectures, such as pre-trained ResNets. We apply these measures to several problems, such as dataset summarization, analysis of under-sampled classes, comparison of informativeness of different data sources, and detection of adversarial and corrupted examples. Our work generalizes existing frameworks but enjoys better computational properties for heavily over-parametrized models, which makes it possible to apply it to real-world networks.

ICLR 2021, 22 pages

References in corpus (13)

• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Understanding Black-box Predictions via Influence Functions
• Opening the Black Box of Deep Neural Networks via Information
• EMNIST: an extension of MNIST to handwritten letters
• Train faster, generalize better: Stability of stochastic gradient descent
• Nonnegative Decomposition of Multivariate Information
• Data Shapley: Equitable Valuation of Data for Machine Learning
• An Empirical Study of Example Forgetting during Deep Neural Network Learning
• iCassava 2019 Fine-Grained Visual Categorization Challenge
• Evaluation of Neural Architectures Trained with Square Loss vs Cross-Entropy in Classification Tasks
• Where is the Information in a Deep Neural Network?
• Influence Functions in Deep Learning Are Fragile
• Information in Infinite Ensembles of Infinitely-Wide Neural Networks