The Resistance to Label Noise in K-NN and DNN Depends on its Concentration

Abstract

We investigate the classification performance of K-nearest neighbors (K-NN) and deep neural networks (DNNs) in the presence of label noise. We first show empirically that a DNN's prediction for a given test example depends on the labels of the training examples in its local neighborhood. This motivates us to derive a realizable analytic expression that approximates the multi-class K-NN classification error in the presence of label noise, which is of independent importance. We then suggest that the expression for K-NN may serve as a first-order approximation for the DNN error. Finally, we demonstrate empirically the proximity of the developed expression to the observed performance of K-NN and DNN classifiers. Our result may explain the already observed surprising resistance of DNN to some types of label noise. It also characterizes an important factor of it showing that the more concentrated the noise the greater is the degradation in performance.

None

References in corpus (16)

• Striving for Simplicity: The All Convolutional Net
• Densely Connected Convolutional Networks
• Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels
• Learning to Reweight Examples for Robust Deep Learning
• DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs
• Training Convolutional Networks with Noisy Labels
• Revisiting Unreasonable Effectiveness of Data in Deep Learning Era
• Dimensionality-Driven Learning with Noisy Labels
• Making Risk Minimization Tolerant to Label Noise
• Masking: A New Perspective of Noisy Supervision
• Robust Loss Functions under Label Noise for Deep Neural Networks
• Learning with Bad Training Data via Iterative Trimmed Loss Minimization
• Joint Optimization Framework for Learning with Noisy Labels
• Learning from Noisy Labels with Distillation
• Learning From Noisy Large-Scale Datasets With Minimal Supervision
• The Devil of Face Recognition is in the Noise

Cited by in corpus (3)

• Constrained Instance and Class Reweighting for Robust Learning under Label Noise
• Multiplicative Reweighting for Robust Neural Network Optimization
• kNet: A Deep kNN Network To Handle Label Noise