Influence Functions in Deep Learning Are Fragile

Abstract

Influence functions approximate the effect of training samples in test-time predictions and have a wide variety of applications in machine learning interpretability and uncertainty estimation. A commonly-used (first-order) influence function can be implemented efficiently as a post-hoc method requiring access only to the gradients and Hessian of the model. For linear models, influence functions are well-defined due to the convexity of the underlying loss function and are generally accurate even across difficult settings where model changes are fairly large such as estimating group influences. Influence functions, however, are not well-understood in the context of deep learning with non-convex loss functions. In this paper, we provide a comprehensive and large-scale empirical study of successes and failures of influence functions in neural network models trained on datasets such as Iris, MNIST, CIFAR-10 and ImageNet. Through our extensive experiments, we show that the network architecture, its depth and width, as well as the extent of model parameterization and regularization techniques have strong effects in the accuracy of influence functions. In particular, we find that (i) influence estimates are fairly accurate for shallow networks, while for deeper networks the estimates are often erroneous; (ii) for certain network architectures and datasets, training with weight-decay regularization is important to get high-quality influence estimates; and (iii) the accuracy of influence estimates can vary significantly depending on the examined test points. These results suggest that in general influence functions in deep learning are fragile and call for developing improved influence estimation methods to mitigate these issues in non-convex setups.

ICLR 2021

References in corpus (12)

• Adam: A Method for Stochastic Optimization
• Machine Learning in Automated Text Categorization
• An overview of gradient descent optimization algorithms
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Understanding Black-box Predictions via Influence Functions
• A disciplined approach to neural network hyper-parameters: Part 1 -- learning rate, batch size, momentum, and weight decay
• Image Segmentation Algorithms Overview
• Estimating Training Data Influence by Tracing Gradient Descent
• Stronger Data Poisoning Attacks Break Data Sanitization Defenses
• On the Accuracy of Influence Functions for Measuring Group Effects
• A Swiss Army Infinitesimal Jackknife
• A Higher-Order Swiss Army Infinitesimal Jackknife

Cited by in corpus (8)

• Adversarial Examples Make Strong Poisons
• Out-of-Distribution Generalization Analysis via Influence Function
• Interactive Label Cleaning with Example-based Explanations
• Estimating informativeness of samples with Smooth Unique Information
• An Empirical Comparison of Instance Attribution Methods for NLP
• Quantifying Epistemic Uncertainty in Deep Learning
• Quantizing data for distributed learning
• ModelPred: A Framework for Predicting Trained Model from Training Data