Fix your classifier: the marginal value of training the last weight layer
arXiv:1801.04540
Abstract
Neural networks are commonly used as models for classification for a wide variety of tasks. Typically, a learned affine transformation is placed at the end of such models, yielding a per-class value used for classification. This classifier can have a vast number of parameters, which grows linearly with the number of possible classes, thus requiring increasingly more resources. In this work we argue that this classifier can be fixed, up to a global scale constant, with little or no loss of accuracy for most tasks, allowing memory and computational benefits. Moreover, we show that by initializing the classifier with a Hadamard matrix we can speed up inference as well. We discuss the implications for current understanding of neural network models.
arXiv commenthttps://openreview.net/forum?id=S1Dh8Tg0-
References in corpus (20)
- MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
- Striving for Simplicity: The All Convolutional Net
- BinaryConnect: Training Deep Neural Networks with binary weights during propagations
- DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
- Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
- Mixed Precision Training
- Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks
- Ternary Weight Networks
- The Marginal Value of Adaptive Gradient Methods in Machine Learning
- Speeding up Convolutional Neural Networks with Low Rank Expansions
- Pointer Sentinel Mixture Models
- Regularizing and Optimizing LSTM Language Models
- Convolutional neural networks with low-rank regularization
- Revisiting Unreasonable Effectiveness of Data in Deep Learning Era
- Exploring Generalization in Deep Learning
- Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modeling
- Spectrally-normalized margin bounds for neural networks
- No bad local minima: Data independent training error guarantees for multilayer neural networks
- Diverse Neural Network Learns True Target Functions
- Exponentially vanishing sub-optimal local minima in multilayer neural networks