NeST: A Neural Network Synthesis Tool Based on a Grow-and-Prune Paradigm

Abstract

Deep neural networks (DNNs) have begun to have a pervasive impact on various applications of machine learning. However, the problem of finding an optimal DNN architecture for large applications is challenging. Common approaches go for deeper and larger DNN architectures but may incur substantial redundancy. To address these problems, we introduce a network growth algorithm that complements network pruning to learn both weights and compact DNN architectures during training. We propose a DNN synthesis tool (NeST) that combines both methods to automate the generation of compact and accurate DNNs. NeST starts with a randomly initialized sparse network called the seed architecture. It iteratively tunes the architecture with gradient-based growth and magnitude-based pruning of neurons and connections. Our experimental results show that NeST yields accurate, yet very compact DNNs, with a wide range of seed architecture selection. For the LeNet-300-100 (LeNet-5) architecture, we reduce network parameters by 70.2x (74.3x) and floating-point operations (FLOPs) by 79.4x (43.7x). For the AlexNet and VGG-16 architectures, we reduce network parameters (FLOPs) by 15.7x (4.6x) and 30.2x (8.6x), respectively. NeST's grow-and-prune paradigm delivers significant additional parameter and FLOPs reduction relative to pruning-only methods.

References in corpus (9)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Sequence to Sequence Learning with Neural Networks
• Neural Architecture Search with Reinforcement Learning
• Compressing Deep Convolutional Networks using Vector Quantization
• One weird trick for parallelizing convolutional neural networks
• Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architectures
• Memory Bounded Deep Convolutional Networks
• Compact Deep Convolutional Neural Networks With Coarse Pruning

Cited by in corpus (11)

• A Systematic DNN Weight Pruning Framework using Alternating Direction Method of Multipliers
• Recurrent Neural Networks: An Embedded Computing Perspective
• Efficient Visual Recognition with Deep Neural Networks: A Survey on Recent Advances and New Directions
• DeepHoyer: Learning Sparser Neural Network with Differentiable Scale-Invariant Sparsity Measures
• CoCoPIE: Making Mobile AI Sweet As PIE --Compression-Compilation Co-Design Goes a Long Way
• ADMM-NN: An Algorithm-Hardware Co-Design Framework of DNNs Using Alternating Direction Method of Multipliers
• Campfire: Compressible, Regularization-Free, Structured Sparse Training for Hardware Accelerators
• AutoGrow: Automatic Layer Growing in Deep Convolutional Networks
• Designing Adaptive Neural Networks for Energy-Constrained Image Classification
• Approximations in Deep Learning
• RePr: Improved Training of Convolutional Filters