Smallify: Learning Network Size while Training

Abstract

As neural networks become widely deployed in different applications and on different hardware, it has become increasingly important to optimize inference time and model size along with model accuracy. Most current techniques optimize model size, model accuracy and inference time in different stages, resulting in suboptimal results and computational inefficiency. In this work, we propose a new technique called Smallify that optimizes all three of these metrics at the same time. Specifically we present a new method to simultaneously optimize network size and model performance by neuron-level pruning during training. Neuron-level pruning not only produces much smaller networks but also produces dense weight matrices that are amenable to efficient inference. By applying our technique to convolutional as well as fully connected models, we show that Smallify can reduce network size by 35X with a 6X improvement in inference time with similar accuracy as models found by traditional training techniques.

arXiv comment11 pages, 3 figures

References in corpus (9)

• Distilling the Knowledge in a Neural Network
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Practical Bayesian Optimization of Machine Learning Algorithms
• Neural Architecture Search with Reinforcement Learning
• FitNets: Hints for Thin Deep Nets
• Compression-aware Training of Deep Networks
• Net-Trim: Convex Pruning of Deep Neural Networks with Performance Guarantee
• Coordinating Filters for Faster Deep Neural Networks
• Nonparametric Neural Networks

Cited by in corpus (4)

• Learning Sparse Networks Using Targeted Dropout
• Neural Network Distiller: A Python Package For DNN Compression Research
• Compressibility Loss for Neural Network Weights
• Towards Efficient Convolutional Network Models with Filter Distribution Templates