Margin-Based Regularization and Selective Sampling in Deep Neural Networks
arXiv:2009.06011
Abstract
We derive a new margin-based regularization formulation, termed multi-margin regularization (MMR), for deep neural networks (DNNs). The MMR is inspired by principles that were applied in margin analysis of shallow linear classifiers, e.g., support vector machine (SVM). Unlike SVM, MMR is continuously scaled by the radius of the bounding sphere (i.e., the maximal norm of the feature vector in the data), which is constantly changing during training. We empirically demonstrate that by a simple supplement to the loss function, our method achieves better results on various classification tasks across domains. Using the same concept, we also derive a selective sampling scheme and demonstrate accelerated training of DNNs by selecting samples according to a minimal margin score (MMS). This score measures the minimal amount of displacement an input should undergo until its predicted classification is switched. We evaluate our proposed methods on three image classification tasks and six language text classification tasks. Specifically, we show improved empirical results on CIFAR10, CIFAR100 and ImageNet using state-of-the-art convolutional neural networks (CNNs) and BERT-BASE architecture for the MNLI, QQP, QNLI, MRPC, SST-2 and RTE benchmarks.
References in corpus (12)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
- Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
- Improved Regularization of Convolutional Neural Networks with Cutout
- Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
- Wide Residual Networks
- AutoAugment: Learning Augmentation Policies from Data
- Robust Large Margin Deep Neural Networks
- Why do deep convolutional networks generalize so poorly to small image transformations?
- Online Batch Selection for Faster Training of Neural Networks
- Not All Samples Are Created Equal: Deep Learning with Importance Sampling
- Variance Reduction in SGD by Distributed Importance Sampling