Understanding Deep Learning via Decision Boundary

Abstract

This paper discovers that the neural network with lower decision boundary (DB) variability has better generalizability. Two new notions, algorithm DB variability and $(\epsilon ,\eta )$-data DB variability, are proposed to measure the decision boundary variability from the algorithm and data perspectives. Extensive experiments show significant negative correlations between the decision boundary variability and the generalizability. From the theoretical view, two lower bounds based on algorithm DB variability are proposed and do not explicitly depend on the sample size. We also prove an upper bound of order $\mathcal{O}\left(\frac{1}{\sqrt{m}}+\epsilon +\eta \log \frac{1}{\eta }\right)$ based on data DB variability. The bound is convenient to estimate without the requirement of labels, and does not explicitly depend on the network size which is usually prohibitively large in deep learning.

arXiv commentAccepted by IEEE TNNLS

References in corpus (26)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Reconciling modern machine learning practice and the bias-variance trade-off
• Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks
• Prevalence of Neural Collapse during the terminal phase of deep learning training
• Train faster, generalize better: Stability of stochastic gradient descent
• How to Escape Saddle Points Efficiently
• Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Training
• The Pitfalls of Simplicity Bias in Neural Networks
• Depth Creates No Bad Local Minima
• Stiffness: A New Perspective on Generalization in Neural Networks
• Towards a Mathematical Understanding of Neural Network-Based Machine Learning: what we know and what we don't
• SGD on Neural Networks Learns Functions of Increasing Complexity
• Characterizing the Decision Boundary of Deep Neural Networks
• Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Loss
• Critical Points of Neural Networks: Analytical Forms and Landscape Properties
• Analysis of Generalizability of Deep Neural Networks Based on the Complexity of Decision Boundary
• Understanding the Decision Boundary of Deep Neural Networks: An Empirical Study
• Recent advances in deep learning theory
• Bayesian Model Selection, the Marginal Likelihood, and Generalization
• Assessing Generalization of SGD via Disagreement
• Piecewise linear activations substantially shape the loss surfaces of neural networks
• Quantifying the Alignment of Graph and Features in Deep Learning
• The Deep Bootstrap Framework: Good Online Learners are Good Offline Generalizers
• Directional convergence and alignment in deep learning
• The Local Elasticity of Neural Networks
• Truth or Backpropaganda? An Empirical Investigation of Deep Learning Theory

Cited by in corpus (1)

• Loss Landscape Topology Reveals Why Simple Baselines are Competitive at 3D Point Cloud Segmentation Under Class Imbalance