Deep Learning using Linear Support Vector Machines

Abstract

Recently, fully-connected and convolutional neural networks have been trained to achieve state-of-the-art performance on a wide variety of tasks such as speech recognition, image classification, natural language processing, and bioinformatics. For classification tasks, most of these "deep learning" models employ the softmax activation function for prediction and minimize cross-entropy loss. In this paper, we demonstrate a small but consistent advantage of replacing the softmax layer with a linear support vector machine. Learning minimizes a margin-based loss instead of the cross-entropy loss. While there have been various combinations of neural nets and SVMs in prior art, our results using L2-SVMs show that by simply replacing softmax with linear SVMs gives significant gains on popular deep learning datasets MNIST, CIFAR-10, and the ICML 2013 Representation Learning Workshop's face expression recognition challenge.

Contribution to the ICML 2013 Challenges in Representation Learning Workshop

References in corpus (2)

• High-Performance Neural Networks for Visual Object Classification
• Adding noise to the input of a model trained with a regularized objective

Cited by in corpus (111)

• Deep Learning using Rectified Linear Units (ReLU)
• AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild
• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• BinaryConnect: Training Deep Neural Networks with binary weights during propagations
• Deep Facial Expression Recognition: A Survey
• Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
• Going Deeper in Facial Expression Recognition using Deep Neural Networks
• Deeply-Supervised Nets
• Towards Deep Neural Network Architectures Robust to Adversarial Examples
• Local Learning with Deep and Handcrafted Features for Facial Expression Recognition
• The Extreme Value Machine
• Recent Advances in Convolutional Neural Networks
• Facial Expression Recognition with Visual Transformers and Attentional Selective Fusion
• Apple Flower Detection using Deep Convolutional Networks
• A Neural Network Architecture Combining Gated Recurrent Unit (GRU) and Support Vector Machine (SVM) for Intrusion Detection in Network Traffic Data
• Analysis of Diffractive Optical Neural Networks and Their Integration with Electronic Neural Networks
• Deep Learning for Human Affect Recognition: Insights and New Developments
• On Breast Cancer Detection: An Application of Machine Learning Algorithms on the Wisconsin Diagnostic Dataset
• Effective Multi-Query Expansions: Collaborative Deep Networks for Robust Landmark Retrieval
• Support vector machines on the D-Wave quantum annealer
• Unsupervised Learning by Competing Hidden Units
• Facial Emotion Recognition: State of the Art Performance on FER2013
• Evaluating Bayesian Deep Learning Methods for Semantic Segmentation
• An Architecture Combining Convolutional Neural Network (CNN) and Support Vector Machine (SVM) for Image Classification
• Mixed-precision deep learning based on computational memory
• Challenges in Representation Learning: A report on three machine learning contests
• Fully Automated Multi-Organ Segmentation in Abdominal Magnetic Resonance Imaging with Deep Neural Networks
• Facial Expression Recognition from World Wild Web
• Artificial Intelligence and its Role in Near Future
• Diving Deep into Sentiment: Understanding Fine-tuned CNNs for Visual Sentiment Prediction
• Towards Building an Intelligent Anti-Malware System: A Deep Learning Approach using Support Vector Machine (SVM) for Malware Classification
• New SAR target recognition based on YOLO and very deep multi-canonical correlation analysis
• Real-time Convolutional Neural Networks for Emotion and Gender Classification
• Multi-Representational Learning for Offline Signature Verification using Multi-Loss Snapshot Ensemble of CNNs
• On Loss Functions for Deep Neural Networks in Classification
• Suppressing Uncertainties for Large-Scale Facial Expression Recognition
• SegNet: A Segmented Deep Learning based Convolutional Neural Network Approach for Drones Wildfire Detection
• Noisy Student Training using Body Language Dataset Improves Facial Expression Recognition
• Individual Recognition in Schizophrenia using Deep Learning Methods with Random Forest and Voting Classifiers: Insights from Resting State EEG Streams
• An Improvement of Data Classification Using Random Multimodel Deep Learning (RMDL)
• Recognizing Semantic Features in Faces using Deep Learning
• Max-Mahalanobis Linear Discriminant Analysis Networks
• Region Attention Networks for Pose and Occlusion Robust Facial Expression Recognition
• Learning Social Relation Traits from Face Images
• OmicsMapNet: Transforming omics data to take advantage of Deep Convolutional Neural Network for discovery
• A Survey of Deep Learning Techniques for Mobile Robot Applications
• A literature survey of matrix methods for data science
• Efficient Gender Classification Using a Deep LDA-Pruned Net
• Max-margin Deep Generative Models
• Convexified Convolutional Neural Networks
• Neural Decision Trees
• Examining the Use of Neural Networks for Feature Extraction: A Comparative Analysis using Deep Learning, Support Vector Machines, and K-Nearest Neighbor Classifiers
• Potential adversarial samples for white-box attacks
• Training Neural Networks for Likelihood/Density Ratio Estimation
• Adversarial Margin Maximization Networks
• Minnorm training: an algorithm for training over-parameterized deep neural networks
• Classification of COVID-19 X-ray Images Using a Combination of Deep and Handcrafted Features
• Frame attention networks for facial expression recognition in videos
• Interactive Deep Refinement Network for Medical Image Segmentation
• Bounded Residual Gradient Networks (BReG-Net) for Facial Affect Computing
• Training Deep Networks for Facial Expression Recognition with Crowd-Sourced Label Distribution
• Deep Learning Neural Networks for Emotion Classification from Text: Enhanced Leaky Rectified Linear Unit Activation and Weighted Loss
• Training Group Orthogonal Neural Networks with Privileged Information
• Fully-Convolutional Intensive Feature Flow Neural Network for Text Recognition
• Support Vector Machine Classifier via $L_{0/1}$ Soft-Margin Loss
• CIFAR-10: KNN-based Ensemble of Classifiers
• GXNOR-Net: Training deep neural networks with ternary weights and activations without full-precision memory under a unified discretization framework
• New Item Consumption Prediction Using Deep Learning
• Deep Low-Density Separation for Semi-Supervised Classification
• Low-rank passthrough neural networks
• Designing GANs: A Likelihood Ratio Approach
• Ensemble Soft-Margin Softmax Loss for Image Classification
• Neural Network Memory Architectures for Autonomous Robot Navigation
• Dense Residual Network: Enhancing Global Dense Feature Flow for Character Recognition
• AU-Guided Unsupervised Domain Adaptive Facial Expression Recognition
• A Regressive Convolution Neural network and Support Vector Regression Model for Electricity Consumption Forecasting
• Deep vs. Diverse Architectures for Classification Problems
• Material Classification using Neural Networks
• Compressed DenseNet for Lightweight Character Recognition
• Reinforcement Learning Based Handwritten Digit Recognition with Two-State Q-Learning
• Winter Road Surface Condition Recognition Using A Pretrained Deep Convolutional Network
• Deep Evolution for Facial Emotion Recognition
• MAFER: a Multi-resolution Approach to Facial Expression Recognition
• ATRO: Adversarial Training with a Rejection Option
• A Survey of Techniques All Classifiers Can Learn from Deep Networks: Models, Optimizations, and Regularization
• Max-Margin Deep Generative Models for (Semi-)Supervised Learning
• Towards the effectiveness of Deep Convolutional Neural Network based Fast Random Forest Classifier
• Face-Cap: Image Captioning using Facial Expression Analysis
• Human Face Expressions from Images - 2D Face Geometry and 3D Face Local Motion versus Deep Neural Features
• Smaller Models, Better Generalization
• Optimizing Shallow Networks for Binary Classification
• SVM-based Deep Stacking Networks
• Neural Conditional Gradients
• Fast Mesh Data Augmentation via Chebyshev Polynomial of Spectral filtering
• Totally Deep Support Vector Machines
• Omni-supervised Facial Expression Recognition via Distilled Data
• Recognizing Facial Expressions in the Wild using Multi-Architectural Representations based Ensemble Learning with Distillation
• Multi-kernel learning of deep convolutional features for action recognition
• Detecting Escalation Level from Speech with Transfer Learning and Acoustic-Lexical Information Fusion
• End-to-end training of deep kernel map networks for image classification
• Biologically Inspired Visual System Architecture for Object Recognition in Autonomous Systems
• Pose-based Body Language Recognition for Emotion and Psychiatric Symptom Interpretation
• A Multi-resolution Approach to Expression Recognition in the Wild
• Human-Centered Emotion Recognition in Animated GIFs
• Recent Advances in Large Margin Learning
• Mixture separability loss in a deep convolutional network for image classification
• Graph Interpolating Activation Improves Both Natural and Robust Accuracies in Data-Efficient Deep Learning
• Unified Backpropagation for Multi-Objective Deep Learning
• A Novel Hybrid Deep Learning Approach for Non-Intrusive Load Monitoring of Residential Appliance Based on Long Short Term Memory and Convolutional Neural Networks
• Manifold Regularized Discriminative Neural Networks
• Graphs for deep learning representations