$L_0$-ARM: Network Sparsification via Stochastic Binary Optimization

Abstract

We consider network sparsification as an $L_0$-norm regularized binary optimization problem, where each unit of a neural network (e.g., weight, neuron, or channel, etc.) is attached with a stochastic binary gate, whose parameters are jointly optimized with original network parameters. The Augment-Reinforce-Merge (ARM), a recently proposed unbiased gradient estimator, is investigated for this binary optimization problem. Compared to the hard concrete gradient estimator from Louizos et al., ARM demonstrates superior performance of pruning network architectures while retaining almost the same accuracies of baseline methods. Similar to the hard concrete estimator, ARM also enables conditional computation during model training but with improved effectiveness due to the exact binary stochasticity. Thanks to the flexibility of ARM, many smooth or non-smooth parametric functions, such as scaled sigmoid or hard sigmoid, can be used to parameterize this binary optimization problem and the unbiasness of the ARM estimator is retained, while the hard concrete estimator has to rely on the hard sigmoid function to achieve conditional computation and thus accelerated training. Extensive experiments on multiple public datasets demonstrate state-of-the-art pruning rates with almost the same accuracies of baseline methods. The resulting algorithm $L_0$-ARM sparsifies the Wide-ResNet models on CIFAR-10 and CIFAR-100 while the hard concrete estimator cannot. The code is public available at https://github.com/leo-yangli/l0-arm.

arXiv commentPublished as a conference paper at ECML 2019

References in corpus (16)

• Adam: A Method for Stochastic Optimization
• Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
• Categorical Reparameterization with Gumbel-Softmax
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
• Learning both Weights and Connections for Efficient Neural Networks
• Learning Structured Sparsity in Deep Neural Networks
• Variational Dropout and the Local Reparameterization Trick
• The State of Sparsity in Deep Neural Networks
• Bayesian Compression for Deep Learning
• Variational Dropout Sparsifies Deep Neural Networks
• REBAR: Low-variance, unbiased gradient estimates for discrete latent variable models
• Structured Bayesian Pruning via Log-Normal Multiplicative Noise
• Backpropagation through the Void: Optimizing control variates for black-box gradient estimation
• ARM: Augment-REINFORCE-Merge Gradient for Stochastic Binary Networks
• Stochastic Variational Optimization

Cited by in corpus (8)

• Resource-Efficient Neural Networks for Embedded Systems
• On the Compression of Neural Networks Using ${\ell }_0$-Norm Regularization and Weight Pruning
• Contextual Dropout: An Efficient Sample-Dependent Dropout Module
• Dynamic Narrowing of VAE Bottlenecks Using GECO and L0 Regularization
• Dirichlet Pruning for Neural Network Compression
• Learning with Multiplicative Perturbations
• Neural Plasticity Networks
• Dep-$L_0$: Improving $L_0$-based Network Sparsification via Dependency Modeling