Channel selection using Gumbel Softmax

Abstract

Important applications such as mobile computing require reducing the computational costs of neural network inference. Ideally, applications would specify their preferred tradeoff between accuracy and speed, and the network would optimize this end-to-end, using classification error to remove parts of the network. Increasing speed can be done either during training - e.g., pruning filters - or during inference - e.g., conditionally executing a subset of the layers. We propose a single end-to-end framework that can improve inference efficiency in both settings. We use a combination of batch activation loss and classification loss, and Gumbel reparameterization to learn network structure. We train end-to-end, and the same technique supports pruning as well as conditional computation. We obtain promising experimental results for ImageNet classification with ResNet (45-52% less computation).

ECCV 2020

References in corpus (15)

• Categorical Reparameterization with Gumbel-Softmax
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• Densely Connected Convolutional Networks
• Pruning Convolutional Neural Networks for Resource Efficient Inference
• AMC: AutoML for Model Compression and Acceleration on Mobile Devices
• The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
• Rethinking the Value of Network Pruning
• Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architectures
• To prune, or not to prune: exploring the efficacy of pruning for model compression
• Adaptive Computation Time for Recurrent Neural Networks
• Variational Dropout Sparsifies Deep Neural Networks
• ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware
• Deep Learning of Representations: Looking Forward
• Gate Decorator: Global Filter Pruning Method for Accelerating Deep Convolutional Neural Networks
• Deciding How to Decide: Dynamic Routing in Artificial Neural Networks