Distribution Density, Tails, and Outliers in Machine Learning: Metrics and Applications

Abstract

We develop techniques to quantify the degree to which a given (training or testing) example is an outlier in the underlying distribution. We evaluate five methods to score examples in a dataset by how well-represented the examples are, for different plausible definitions of "well-represented", and apply these to four common datasets: MNIST, Fashion-MNIST, CIFAR-10, and ImageNet. Despite being independent approaches, we find all five are highly correlated, suggesting that the notion of being well-represented can be quantified. Among other uses, we find these methods can be combined to identify (a) prototypical examples (that match human expectations); (b) memorized training examples; and, (c) uncommon submodes of the dataset. Further, we show how we can utilize our metrics to determine an improved ordering for curriculum learning, and impact adversarial robustness. We release all metric values on training and test sets we studied.

References in corpus (11)

• Deep Learning with Differential Privacy
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Intriguing properties of neural networks
• Learning to Reweight Examples for Robust Deep Learning
• MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labels
• Prototype selection for interpretable classification
• Deep Learning for Case-Based Reasoning through Prototypes: A Neural Network that Explains Its Predictions
• Biased Importance Sampling for Deep Neural Network Training
• ConvNets and ImageNet Beyond Accuracy: Understanding Mistakes and Uncovering Biases
• Data Dropout: Optimizing Training Data for Convolutional Neural Networks
• Generic Coreset for Scalable Learning of Monotonic Kernels: Logistic Regression, Sigmoid and more

Cited by in corpus (5)

• FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence
• What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimation
• Deep Learning Through the Lens of Example Difficulty
• A Tale Of Two Long Tails
• Towards the Memorization Effect of Neural Networks in Adversarial Training