On Loss Functions for Deep Neural Networks in Classification

Abstract

Deep neural networks are currently among the most commonly used classifiers. Despite easily achieving very good performance, one of the best selling points of these models is their modular design - one can conveniently adapt their architecture to specific needs, change connectivity patterns, attach specialised layers, experiment with a large amount of activation functions, normalisation schemes and many others. While one can find impressively wide spread of various configurations of almost every aspect of the deep nets, one element is, in authors' opinion, underrepresented - while solving classification problems, vast majority of papers and applications simply use log loss. In this paper we try to investigate how particular choices of loss functions affect deep models and their learning dynamics, as well as resulting classifiers robustness to various effects. We perform experiments on classical datasets, as well as provide some additional, theoretical insights into the problem. In particular we show that L1 and L2 losses are, quite surprisingly, justified classification objectives for deep nets, by providing probabilistic interpretation in terms of expected misclassification. We also introduce two losses which are not typically used as deep nets objectives and show that they are viable alternatives to the existing ones.

Presented at Theoretical Foundations of Machine Learning 2017 (TFML 2017)

References in corpus (3)

• Adam: A Method for Stochastic Optimization
• Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)
• Deep Learning using Linear Support Vector Machines

Cited by in corpus (23)

• A Review on Deep Learning Techniques for Video Prediction
• Shape and Margin-Aware Lung Nodule Classification in Low-dose CT Images via Soft Activation Mapping
• Rethinking multiscale cardiac electrophysiology with machine learning and predictive modelling
• Evaluation of Neural Architectures Trained with Square Loss vs Cross-Entropy in Classification Tasks
• A Non-Technical Survey on Deep Convolutional Neural Network Architectures
• Feature Intertwiner for Object Detection
• Training Neural Networks for Likelihood/Density Ratio Estimation
• Smart-PGSim: Using Neural Network to Accelerate AC-OPF Power Grid Simulation
• Convolutional Normalization: Improving Deep Convolutional Network Robustness and Training
• MyDigitalFootprint: an extensive context dataset for pervasive computing applications at the edge
• ETH-DS3Lab at SemEval-2018 Task 7: Effectively Combining Recurrent and Convolutional Neural Networks for Relation Classification and Extraction
• ReActNet: Temporal Localization of Repetitive Activities in Real-World Videos
• The Traveling Observer Model: Multi-task Learning Through Spatial Variable Embeddings
• On the Importance of 3D Surface Information for Remote Sensing Classification Tasks
• The Bregman-Tweedie Classification Model
• Optimizing Shallow Networks for Binary Classification
• Optimal Transfer Learning Model for Binary Classification of Funduscopic Images through Simple Heuristics
• "You eat with your eyes first": Optimizing Yelp Image Advertising
• Nonlinear Collaborative Scheme for Deep Neural Networks
• Prediction of Hereditary Cancers Using Neural Networks
• A Sparsity Algorithm with Applications to Corporate Credit Rating
• Mixing between the Cross Entropy and the Expectation Loss Terms
• Towards end-to-end pulsed eddy current classification and regression with CNN