Large Margin Deep Networks for Classification

Abstract

We present a formulation of deep learning that aims at producing a large margin classifier. The notion of margin, minimum distance to a decision boundary, has served as the foundation of several theoretically profound and empirically successful results for both classification and regression tasks. However, most large margin algorithms are applicable only to shallow models with a preset feature representation; and conventional margin methods for neural networks only enforce margin at the output layer. Such methods are therefore not well suited for deep networks. In this work, we propose a novel loss function to impose a margin on any chosen set of layers of a deep network (including input and hidden layers). Our formulation allows choosing any norm on the metric measuring the margin. We demonstrate that the decision boundary obtained by our loss has nice properties compared to standard classification loss functions. Specifically, we show improved empirical results on the MNIST, CIFAR-10 and ImageNet datasets on multiple tasks: generalization from small training sets, corrupted labels, and robustness against adversarial perturbations. The resulting loss is general and complementary to existing data augmentation (such as random/adversarial input transform) and regularization techniques (such as weight decay, dropout, and batch norm).

References in corpus (15)

• Intriguing properties of neural networks
• Wide Residual Networks
• Matching Networks for One Shot Learning
• Synthesizing Robust Adversarial Examples
• Semi-Supervised Learning with Ladder Networks
• Deep Bayesian Active Learning with Image Data
• Training Deep Neural Networks on Noisy Labels with Bootstrapping
• Training Convolutional Networks with Noisy Labels
• Countering Adversarial Images using Input Transformations
• Large-Margin Softmax Loss for Convolutional Neural Networks
• Learning with a Strong Adversary
• Robust Large Margin Deep Neural Networks
• Parseval Networks: Improving Robustness to Adversarial Examples
• Universal adversarial perturbations
• Google's Cloud Vision API Is Not Robust To Noise

Cited by in corpus (8)

• Predicting the Generalization Gap in Deep Networks with Margin Distributions
• Max-margin Class Imbalanced Learning with Gaussian Affinity
• Built-in Vulnerabilities to Imperceptible Adversarial Perturbations
• RecurJac: An Efficient Recursive Algorithm for Bounding Jacobian Matrix of Neural Networks and Its Applications
• The Best of Both Worlds: a Framework for Combining Degradation Prediction with High Performance Super-Resolution Networks
• Noisy Feature Mixup
• Contrast R-CNN for Continual Learning in Object Detection
• Bridged Adversarial Training