Recent Advances in Large Margin Learning

Abstract

This paper serves as a survey of recent advances in large margin training and its theoretical foundations, mostly for (nonlinear) deep neural networks (DNNs) that are probably the most prominent machine learning models for large-scale data in the community over the past decade. We generalize the formulation of classification margins from classical research to latest DNNs, summarize theoretical connections between the margin, network generalization, and robustness, and introduce recent efforts in enlarging the margins for DNNs comprehensively. Since the viewpoint of different methods is discrepant, we categorize them into groups for ease of comparison and discussion in the paper. Hopefully, our discussions and overview inspire new research work in the community that aim to improve the performance of DNNs, and we also point to directions where the large margin principle can be verified to provide theoretical evidence why certain regularizations for DNNs function well in practice. We managed to shorten the paper such that the crucial spirit of large margin learning and related methods are better emphasized.

arXiv commentAccepted by TPAMI, 8 pages, 3 figures

References in corpus (38)

• Improving neural networks by preventing co-adaptation of feature detectors
• mixup: Beyond Empirical Risk Minimization
• Improved Regularization of Convolutional Neural Networks with Cutout
• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Deep Learning using Linear Support Vector Machines
• Provable defenses against adversarial examples via the convex outer adversarial polytope
• Certified Adversarial Robustness via Randomized Smoothing
• On Exact Computation with an Infinitely Wide Neural Net
• Efficient Neural Network Robustness Certification with General Activation Functions
• Evaluating Robustness of Neural Networks with Mixed Integer Programming
• Robustness and Regularization of Support Vector Machines
• A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networks
• Sensitivity and Generalization in Neural Networks: an Empirical Study
• Improving the Adversarial Robustness and Interpretability of Deep Neural Networks by Regularizing their Input Gradients
• Formal Guarantees on the Robustness of a Classifier against Adversarial Manipulation
• Robust Large Margin Deep Neural Networks
• Spectral Norm Regularization for Improving the Generalizability of Deep Learning
• Measuring Catastrophic Forgetting in Neural Networks
• Lipschitz-Margin Training: Scalable Certification of Perturbation Invariance for Deep Neural Networks
• DSD: Dense-Sparse-Dense Training for Deep Neural Networks
• Gradient descent aligns the layers of deep linear networks
• Sparse DNNs with Improved Adversarial Robustness
• Deep Defense: Training DNNs with Improved Adversarial Robustness
• Regularization Matters: Generalization and Optimization of Neural Nets v.s. their Induced Kernel
• SGD Learns the Conjugate Kernel Class of the Network
• Provable Robustness of ReLU networks via Maximization of Linear Regions
• Data-dependent Sample Complexity of Deep Neural Networks via Lipschitz Augmentation
• Support Vector Guided Softmax Loss for Face Recognition
• Generalization Error of Invariant Classifiers
• Virtual Class Enhanced Discriminative Embedding Learning
• Improved Sample Complexities for Deep Networks and Robust Classification via an All-Layer Margin
• SupportNet: solving catastrophic forgetting in class incremental learning with support data
• Large Margin Few-Shot Learning
• Towards Robust, Locally Linear Deep Networks
• Adversarial Margin Maximization Networks
• Convergence and Margin of Adversarial Training on Separable Data
• Understanding Adversarial Robustness: The Trade-off between Minimum and Average Margin