Stochastic Downsampling for Cost-Adjustable Inference and Improved Regularization in Convolutional Networks

Abstract

It is desirable to train convolutional networks (CNNs) to run more efficiently during inference. In many cases however, the computational budget that the system has for inference cannot be known beforehand during training, or the inference budget is dependent on the changing real-time resource availability. Thus, it is inadequate to train just inference-efficient CNNs, whose inference costs are not adjustable and cannot adapt to varied inference budgets. We propose a novel approach for cost-adjustable inference in CNNs - Stochastic Downsampling Point (SDPoint). During training, SDPoint applies feature map downsampling to a random point in the layer hierarchy, with a random downsampling ratio. The different stochastic downsampling configurations known as SDPoint instances (of the same model) have computational costs different from each other, while being trained to minimize the same prediction loss. Sharing network parameters across different instances provides significant regularization boost. During inference, one may handpick a SDPoint instance that best fits the inference budget. The effectiveness of SDPoint, as both a cost-adjustable inference approach and a regularizer, is validated through extensive experiments on image classification.

References in corpus (21)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Deep Residual Learning for Image Recognition
• Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition
• Wide Residual Networks
• Going Deeper with Convolutions
• Binarized Neural Networks
• Trained Ternary Quantization
• FractalNet: Ultra-Deep Neural Networks without Residuals
• Speeding up Convolutional Neural Networks with Low Rank Expansions
• YOLO9000: Better, Faster, Stronger
• Aggregated Residual Transformations for Deep Neural Networks
• Xception: Deep Learning with Depthwise Separable Convolutions
• Channel Pruning for Accelerating Very Deep Neural Networks
• Adaptive Computation Time for Recurrent Neural Networks
• Systematic evaluation of CNN advances on the ImageNet
• Revisiting Unreasonable Effectiveness of Data in Deep Learning Era
• Deep Networks with Stochastic Depth
• Swapout: Learning an ensemble of deep architectures
• Memory-Efficient Implementation of DenseNets
• BranchyNet: Fast Inference via Early Exiting from Deep Neural Networks
• Spatially Adaptive Computation Time for Residual Networks

Cited by in corpus (4)

• Slimmable Neural Networks
• Once-for-All: Train One Network and Specialize it for Efficient Deployment
• Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation
• InstaNAS: Instance-aware Neural Architecture Search