DNN or k-NN: That is the Generalize vs. Memorize Question

Abstract

This paper studies the relationship between the classification performed by deep neural networks (DNNs) and the decision of various classical classifiers, namely k-nearest neighbours (k-NN), support vector machines (SVM) and logistic regression (LR), at various layers of the network. This comparison provides us with new insights as to the ability of neural networks to both memorize the training data and generalize to new data at the same time, where k-NN serves as the ideal estimator that perfectly memorizes the data. We show that memorization of non-generalizing networks happens only at the last layers. Moreover, the behavior of DNNs compared to the linear classifiers SVM and LR is quite the same on the training and test data regardless of whether the network generalizes. On the other hand, the similarity to k-NN holds only at the absence of overfitting. Our results suggests that k-NN behavior of the network on new data is a sign of generalization. Moreover, it shows that memorization and generalization, which are traditionally considered to be contradicting to each other, are compatible and complementary.

arXiv commentPoster presented in NIPS 2018 "Integration of Deep Learning Theories" workshop

References in corpus (11)

• Wide Residual Networks
• Deep k-Nearest Neighbors: Towards Confident, Interpretable and Robust Deep Learning
• A Closer Look at Memorization in Deep Networks
• Revisiting Unreasonable Effectiveness of Data in Deep Learning Era
• Wide & Deep Learning for Recommender Systems
• Robust Large Margin Deep Neural Networks
• Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data
• Stronger generalization bounds for deep nets via a compression approach
• Significance of Softmax-based Features in Comparison to Distance Metric Learning-based Features
• Theory of Deep Learning III: explaining the non-overfitting puzzle
• A Bayesian Perspective on Generalization and Stochastic Gradient Descent

Cited by in corpus (4)

• What Do Neural Networks Learn When Trained With Random Labels?
• Neural Network Memorization Dissection
• The Impact of Reinitialization on Generalization in Convolutional Neural Networks
• kNet: A Deep kNN Network To Handle Label Noise