Rethinking Weight Decay For Efficient Neural Network Pruning

Abstract

Introduced in the late 1980s for generalization purposes, pruning has now become a staple for compressing deep neural networks. Despite many innovations in recent decades, pruning approaches still face core issues that hinder their performance or scalability. Drawing inspiration from early work in the field, and especially the use of weight decay to achieve sparsity, we introduce Selective Weight Decay (SWD), which carries out efficient, continuous pruning throughout training. Our approach, theoretically grounded on Lagrangian smoothing, is versatile and can be applied to multiple tasks, networks, and pruning structures. We show that SWD compares favorably to state-of-the-art approaches, in terms of performance-to-parameters ratio, on the CIFAR-10, Cora, and ImageNet ILSVRC2012 datasets.

arXiv comment23 pages, 18 figures, published at Journal of Imaging, update : added new results, rewrite

References in corpus (35)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Distilling the Knowledge in a Neural Network
• Semi-Supervised Classification with Graph Convolutional Networks
• BinaryConnect: Training Deep Neural Networks with binary weights during propagations
• Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
• Pruning Convolutional Neural Networks for Resource Efficient Inference
• The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
• Rethinking the Value of Network Pruning
• Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architectures
• SNIP: Single-shot Network Pruning based on Connection Sensitivity
• Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Science
• The State of Sparsity in Deep Neural Networks
• Bayesian Compression for Deep Learning
• Variational Dropout Sparsifies Deep Neural Networks
• Sparse Networks from Scratch: Faster Training without Losing Performance
• Comparing Rewinding and Fine-tuning in Neural Network Pruning
• Deep Rewiring: Training very sparse deep networks
• Learning Sparse Neural Networks through $L_0$ Regularization
• Picking Winning Tickets Before Training by Preserving Gradient Flow
• Stabilizing the Lottery Ticket Hypothesis
• Dynamic Network Surgery for Efficient DNNs
• Structured Bayesian Pruning via Log-Normal Multiplicative Noise
• Compressing Neural Networks using the Variational Information Bottleneck
• Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterization
• Rigging the Lottery: Making All Tickets Winners
• Deconstructing Lottery Tickets: Zeros, Signs, and the Supermask
• One ticket to win them all: generalizing lottery ticket initializations across datasets and optimizers
• Linear Mode Connectivity and the Lottery Ticket Hypothesis
• Proving the Lottery Ticket Hypothesis: Pruning is All You Need
• Learning to Prune Deep Neural Networks via Layer-wise Optimal Brain Surgeon
• Compact Deep Convolutional Neural Networks With Coarse Pruning
• PCAS: Pruning Channels with Attention Statistics for Deep Network Compression
• Non-Structured DNN Weight Pruning -- Is It Beneficial in Any Platform?

Cited by in corpus (3)

• Effective Multi-Stage Training Model For Edge Computing Devices In Intrusion Detection
• Leveraging Structured Pruning of Convolutional Neural Networks
• Energy Consumption Analysis of pruned Semantic Segmentation Networks on an Embedded GPU