Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterization

Abstract

Modern deep neural networks are typically highly overparameterized. Pruning techniques are able to remove a significant fraction of network parameters with little loss in accuracy. Recently, techniques based on dynamic reallocation of non-zero parameters have emerged, allowing direct training of sparse networks without having to pre-train a large dense model. Here we present a novel dynamic sparse reparameterization method that addresses the limitations of previous techniques such as high computational cost and the need for manual configuration of the number of free parameters allocated to each layer. We evaluate the performance of dynamic reallocation methods in training deep convolutional networks and show that our method outperforms previous static and dynamic reparameterization methods, yielding the best accuracy for a fixed parameter budget, on par with accuracies obtained by iteratively pruning a pre-trained dense model. We further investigated the mechanisms underlying the superior generalization performance of the resultant sparse networks. We found that neither the structure, nor the initialization of the non-zero parameters were sufficient to explain the superior performance. Rather, effective learning crucially depended on the continuous exploration of the sparse network structure space during training. Our work suggests that exploring structural degrees of freedom during training is more effective than adding extra parameters to the network.

Proceedings of the 36th International Conference on MachineLearning, Long Beach, California, PMLR 97, 2019

Cited by in corpus (60)

• On the Opportunities and Risks of Foundation Models
• Sparse Networks from Scratch: Faster Training without Losing Performance
• Picking Winning Tickets Before Training by Preserving Gradient Flow
• Rigging the Lottery: Making All Tickets Winners
• Soft Threshold Weight Reparameterization for Learnable Sparsity
• Dynamic Model Pruning with Feedback
• Chasing Sparsity in Vision Transformers: An End-to-End Exploration
• TensorDash: Exploiting Sparsity to Accelerate Deep Neural Network Training and Inference
• Pruning neural networks without any data by iteratively conserving synaptic flow
• Learning N:M Fine-grained Structured Sparse Neural Networks From Scratch
• SpaceNet: Make Free Space For Continual Learning
• BASE Layers: Simplifying Training of Large, Sparse Models
• MEST: Accurate and Fast Memory-Economic Sparse Training Framework on the Edge
• Progressive Skeletonization: Trimming more fat from a network at initialization
• WoodFisher: Efficient Second-Order Approximation for Neural Network Compression
• Low-Memory Neural Network Training: A Technical Report
• The Difficulty of Training Sparse Neural Networks
• Rethinking Weight Decay For Efficient Neural Network Pruning
• Dynamic Sparse Training: Find Efficient Sparse Network From Scratch With Trainable Masked Layers
• CHIP: CHannel Independence-based Pruning for Compact Neural Networks
• A Brain-inspired Algorithm for Training Highly Sparse Neural Networks
• Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Training
• Keep the Gradients Flowing: Using Gradient Flow to Study Sparse Network Optimization
• Can Subnetwork Structure be the Key to Out-of-Distribution Generalization?
• Selfish Sparse RNN Training
• Efficient Neural Network Training via Forward and Backward Propagation Sparsification
• Accelerated Sparse Neural Training: A Provable and Efficient Method to Find N:M Transposable Masks
• Sparse Training via Boosting Pruning Plasticity with Neuroregeneration
• Towards Low-Latency Energy-Efficient Deep SNNs via Attention-Guided Compression
• Deep Ensembling with No Overhead for either Training or Testing: The All-Round Blessings of Dynamic Sparsity
• Sparse Weight Activation Training
• Campfire: Compressible, Regularization-Free, Structured Sparse Training for Hardware Accelerators
• Kaleidoscope: An Efficient, Learnable Representation For All Structured Linear Maps
• Powerpropagation: A sparsity inducing weight reparameterisation
• word2ket: Space-efficient Word Embeddings inspired by Quantum Entanglement
• Activation function impact on Sparse Neural Networks
• On improving deep learning generalization with adaptive sparse connectivity
• Sparse evolutionary Deep Learning with over one million artificial neurons on commodity hardware
• Consistent Sparse Deep Learning: Theory and Computation
• Sparse Deep Learning: A New Framework Immune to Local Traps and Miscalibration
• Effective Model Sparsification by Scheduled Grow-and-Prune Methods
• Pruning Randomly Initialized Neural Networks with Iterative Randomization
• Effective Sparsification of Neural Networks with Global Sparsity Constraint
• Connectivity Matters: Neural Network Pruning Through the Lens of Effective Sparsity
• Topological Insights into Sparse Neural Networks
• Mesa: A Memory-saving Training Framework for Transformers
• Lottery Jackpots Exist in Pre-trained Models
• Dynamic Collective Intelligence Learning: Finding Efficient Sparse Model via Refined Gradients for Pruned Weights
• Initialization and Regularization of Factorized Neural Layers
• ESPN: Extremely Sparse Pruned Networks
• The Elastic Lottery Ticket Hypothesis
• Training highly effective connectivities within neural networks with randomly initialized, fixed weights
• Livewired Neural Networks: Making Neurons That Fire Together Wire Together
• Distributed Sparse SGD with Majority Voting
• Cross-Channel Intragroup Sparsity Neural Network
• On the Downstream Performance of Compressed Word Embeddings
• Architecture Aware Latency Constrained Sparse Neural Networks
• Learning Pruned Structure and Weights Simultaneously from Scratch: an Attention based Approach
• Procrustes: a Dataflow and Accelerator for Sparse Deep Neural Network Training
• Dense for the Price of Sparse: Improved Performance of Sparsely Initialized Networks via a Subspace Offset