From ImageNet to Image Classification: Contextualizing Progress on Benchmarks

Abstract

Building rich machine learning datasets in a scalable manner often necessitates a crowd-sourced data collection pipeline. In this work, we use human studies to investigate the consequences of employing such a pipeline, focusing on the popular ImageNet dataset. We study how specific design choices in the ImageNet creation process impact the fidelity of the resulting dataset---including the introduction of biases that state-of-the-art models exploit. Our analysis pinpoints how a noisy data collection pipeline can lead to a systematic misalignment between the resulting benchmark and the real-world task it serves as a proxy for. Finally, our findings emphasize the need to augment our current model training and evaluation toolkit to take such misalignments into account. To facilitate further research, we release our refined ImageNet annotations at https://github.com/MadryLab/ImageNetMultiLabel.

Cited by in corpus (19)

• Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks
• It's COMPASlicated: The Messy Relationship between RAI Datasets and Algorithmic Fairness Benchmarks
• When and How to Fool Explainable Models (and Humans) with Adversarial Examples
• How Accurate Does It Feel? -- Human Perception of Different Types of Classification Mistakes
• How Well do Feature Visualizations Support Causal Understanding of CNN Activations?
• BREEDS: Benchmarks for Subpopulation Shift
• Reduced, Reused and Recycled: The Life of a Dataset in Machine Learning Research
• Did the Model Change? Efficiently Assessing Machine Learning API Shifts
• Re-labeling ImageNet: from Single to Multi-Labels, from Global to Localized Labels
• Multi-label Iterated Learning for Image Classification with Label Ambiguity
• Leveraging Sparse Linear Layers for Debuggable Deep Networks
• When does loss-based prioritization fail?
• Semantically Adversarial Learnable Filters
• Trivial or impossible -- dichotomous data difficulty masks model differences (on ImageNet and beyond)
• A Tale Of Two Long Tails
• Automated Cleanup of the ImageNet Dataset by Model Consensus, Explainability and Confident Learning
• Who Decides if AI is Fair? The Labels Problem in Algorithmic Auditing
• Disentangled Variational Information Bottleneck for Multiview Representation Learning
• Active Learning at the ImageNet Scale