FLOPs as a Direct Optimization Objective for Learning Sparse Neural Networks

Abstract

There exists a plethora of techniques for inducing structured sparsity in parametric models during the optimization process, with the final goal of resource-efficient inference. However, few methods target a specific number of floating-point operations (FLOPs) as part of the optimization objective, despite many reporting FLOPs as part of the results. Furthermore, a one-size-fits-all approach ignores realistic system constraints, which differ significantly between, say, a GPU and a mobile phone -- FLOPs on the former incur less latency than on the latter; thus, it is important for practitioners to be able to specify a target number of FLOPs during model compression. In this work, we extend a state-of-the-art technique to directly incorporate FLOPs as part of the optimization objective and show that, given a desired FLOPs requirement, different neural networks can be successfully trained for image classification.

arXiv comment4 pages, accepted to the NIPS 2018 Workshop on Compact Deep Neural Networks with Industrial Applications (CDNNRIA)

References in corpus (5)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Wide Residual Networks
• Bayesian Compression for Deep Learning
• An Experimental Analysis of the Power Consumption of Convolutional Neural Networks for Keyword Spotting

Cited by in corpus (5)

• Distilling Task-Specific Knowledge from BERT into Simple Neural Networks
• Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Search
• Radial and Directional Posteriors for Bayesian Neural Networks
• Neuron ranking -- an informed way to condense convolutional neural networks architecture
• Dirichlet Pruning for Neural Network Compression