Learning Strict Identity Mappings in Deep Residual Networks
arXiv:1804.01661
Abstract
A family of super deep networks, referred to as residual networks or ResNet, achieved record-beating performance in various visual tasks such as image recognition, object detection, and semantic segmentation. The ability to train very deep networks naturally pushed the researchers to use enormous resources to achieve the best performance. Consequently, in many applications super deep residual networks were employed for just a marginal improvement in performance. In this paper, we propose epsilon-ResNet that allows us to automatically discard redundant layers, which produces responses that are smaller than a threshold epsilon, with a marginal or no loss in performance. The epsilon-ResNet architecture can be achieved using a few additional rectified linear units in the original ResNet. Our method does not use any additional variables nor numerous trials like other hyper-parameter optimization techniques. The layer selection is achieved using a single training process and the evaluation is performed on CIFAR-10, CIFAR-100, SVHN, and ImageNet datasets. In some instances, we achieve about 80% reduction in the number of parameters.
arXiv commentMake title consistent with the CVPR version
References in corpus (17)
- MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
- SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
- Practical Bayesian Optimization of Machine Learning Algorithms
- Neural Architecture Search with Reinforcement Learning
- R-FCN: Object Detection via Region-based Fully Convolutional Networks
- Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
- Pruning Convolutional Neural Networks for Resource Efficient Inference
- On the Number of Linear Regions of Deep Neural Networks
- Highway Networks
- Large-Scale Evolution of Image Classifiers
- Some Improvements on Deep Convolutional Neural Network Based Image Classification
- No More Pesky Learning Rates
- Practical recommendations for gradient-based training of deep architectures
- Wider or Deeper: Revisiting the ResNet Model for Visual Recognition
- Learning the Number of Neurons in Deep Networks
- Bounding and Counting Linear Regions of Deep Neural Networks
- Data-Driven Sparse Structure Selection for Deep Neural Networks