Batch-Shaping for Learning Conditional Channel Gated Networks

Abstract

We present a method that trains large capacity neural networks with significantly improved accuracy and lower dynamic computational cost. We achieve this by gating the deep-learning architecture on a fine-grained-level. Individual convolutional maps are turned on/off conditionally on features in the network. To achieve this, we introduce a new residual block architecture that gates convolutional channels in a fine-grained manner. We also introduce a generally applicable tool $batch$-$shaping$ that matches the marginal aggregate posteriors of features in a neural network to a pre-specified prior distribution. We use this novel technique to force gates to be more conditional on the data. We present results on CIFAR-10 and ImageNet datasets for image classification, and Cityscapes for semantic segmentation. Our results show that our method can slim down large architectures conditionally, such that the average computational cost on the data is on par with a smaller architecture, but with higher accuracy. In particular, on ImageNet, our ResNet50 and ResNet34 gated networks obtain 74.60% and 72.55% top-1 accuracy compared to the 69.76% accuracy of the baseline ResNet18 model, for similar complexity. We also show that the resulting networks automatically learn to use more features for difficult examples and fewer features for simple examples.

Published as a conference paper at ICLR 2020

References in corpus (11)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Pruning Convolutional Neural Networks for Resource Efficient Inference
• The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
• Speeding up Convolutional Neural Networks with Low Rank Expansions
• Bayesian Compression for Deep Learning
• Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
• Multi-Scale Dense Networks for Resource Efficient Image Classification
• Dynamic Channel Pruning: Feature Boosting and Suppression
• Compressing Neural Networks using the Variational Information Bottleneck
• Gated networks: an inventory
• SGAD: Soft-Guided Adaptively-Dropped Neural Network

Cited by in corpus (8)

• Dynamic Neural Networks: A Survey
• SegBlocks: Block-Based Dynamic Resolution Networks for Real-Time Segmentation
• AdaViT: Adaptive Vision Transformers for Efficient Image Recognition
• Generative Zero-shot Network Quantization
• 2D or not 2D? Adaptive 3D Convolution Selection for Efficient Video Recognition
• TimeGate: Conditional Gating of Segments in Long-range Activities
• Prioritized Subnet Sampling for Resource-Adaptive Supernet Training
• Human-Object Interaction Detection via Weak Supervision