Cold Case: The Lost MNIST Digits

Abstract

Although the popular MNIST dataset [LeCun et al., 1994] is derived from the NIST database [Grother and Hanaoka, 1995], the precise processing steps for this derivation have been lost to time. We propose a reconstruction that is accurate enough to serve as a replacement for the MNIST dataset, with insignificant changes in accuracy. We trace each MNIST digit to its NIST source and its rich metadata such as writer identifier, partition identifier, etc. We also reconstruct the complete MNIST test set with 60,000 samples instead of the usual 10,000. Since the balance 50,000 were never distributed, they enable us to investigate the impact of twenty-five years of MNIST experiments on the reported testing performances. Our results unambiguously confirm the trends observed by Recht et al. [2018, 2019]: although the misclassification rates are slightly off, classifier ordering and model selection remain broadly reliable. We attribute this phenomenon to the pairing benefits of comparing classifiers on the same digits.

arXiv commentFinal NeurIPS version

References in corpus (4)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Do ImageNet Classifiers Generalize to ImageNet?
• Do CIFAR-10 Classifiers Generalize to CIFAR-10?
• The advantages of multiple classes for reducing overfitting from test set reuse

Cited by in corpus (20)

• SpinalNet: Deep Neural Network with Gradual Input
• Measuring Robustness in Deep Learning Based Compressive Sensing
• Robust Validation: Confident Predictions Even When Distributions Shift
• Evaluating Models' Local Decision Boundaries via Contrast Sets
• The worst of both worlds: A comparative analysis of errors in learning from data in psychology and machine learning
• Deep regularization and direct training of the inner layers of Neural Networks with Kernel Flows
• The Evolution of Out-of-Distribution Robustness Throughout Fine-Tuning
• Morpho-MNIST: Quantitative Assessment and Diagnostics for Representation Learning
• That which we call private
• Disentangling Style and Content in Anime Illustrations
• The Benchmark Lottery
• Detecting Overfitting via Adversarial Examples
• Tensor network to learn the wavefunction of data
• A Spike in Performance: Training Hybrid-Spiking Neural Networks with Quantized Activation Functions
• Model-Based Robust Deep Learning: Generalizing to Natural, Out-of-Distribution Data
• OmniPrint: A Configurable Printed Character Synthesizer
• Domain2Vec: Domain Embedding for Unsupervised Domain Adaptation
• Optimal multiclass overfitting by sequence reconstruction from Hamming queries
• A Makefile for Developing Containerized LaTeX Technical Documents
• Tightening the Approximation Error of Adversarial Risk with Auto Loss Function Search