On the Decision Boundary of Deep Neural Networks

Abstract

While deep learning models and techniques have achieved great empirical success, our understanding of the source of success in many aspects remains very limited. In an attempt to bridge the gap, we investigate the decision boundary of a production deep learning architecture with weak assumptions on both the training data and the model. We demonstrate, both theoretically and empirically, that the last weight layer of a neural network converges to a linear SVM trained on the output of the last hidden layer, for both the binary case and the multi-class case with the commonly used cross-entropy loss. Furthermore, we show empirically that training a neural network as a whole, instead of only fine-tuning the last weight layer, may result in better bias constant for the last weight layer, which is important for generalization. In addition to facilitating the understanding of deep learning, our result can be helpful for solving a broad range of practical problems of deep learning, such as catastrophic forgetting and adversarial attacking. The experiment codes are available at https://github.com/lykaust15/NN_decision_boundary

References in corpus (13)

• Overcoming catastrophic forgetting in neural networks
• Natural Language Processing (almost) from Scratch
• How transferable are features in deep neural networks?
• Understanding deep learning requires rethinking generalization
• The Loss Surfaces of Multilayer Networks
• The loss surface of deep and wide neural networks
• Recovery Guarantees for One-hidden-layer Neural Networks
• Stronger generalization bounds for deep nets via a compression approach
• Exponentially vanishing sub-optimal local minima in multilayer neural networks
• Theory of Deep Learning III: explaining the non-overfitting puzzle
• Theory II: Landscape of the Empirical Risk in Deep Learning
• SGD Learns Over-parameterized Networks that Provably Generalize on Linearly Separable Data
• SupportNet: solving catastrophic forgetting in class incremental learning with support data

Cited by in corpus (9)

• Accelerating Flash Calculation through Deep Learning Methods
• Characterizing the Decision Boundary of Deep Neural Networks
• Analysis of Generalizability of Deep Neural Networks Based on the Complexity of Decision Boundary
• SupportNet: solving catastrophic forgetting in class incremental learning with support data
• Hold me tight! Influence of discriminative features on deep network boundaries
• Deep learning in bioinformatics: introduction, application, and perspective in big data era
• Neural Language Priors
• A Tour of Convolutional Networks Guided by Linear Interpreters
• How to Explain Neural Networks: an Approximation Perspective