Powerpropagation: A sparsity inducing weight reparameterisation

Abstract

The training of sparse neural networks is becoming an increasingly important tool for reducing the computational footprint of models at training and evaluation, as well enabling the effective scaling up of models. Whereas much work over the years has been dedicated to specialised pruning techniques, little attention has been paid to the inherent effect of gradient based training on model sparsity. In this work, we introduce Powerpropagation, a new weight-parameterisation for neural networks that leads to inherently sparse models. Exploiting the behaviour of gradient descent, our method gives rise to weight updates exhibiting a "rich get richer" dynamic, leaving low-magnitude parameters largely unaffected by learning. Models trained in this manner exhibit similar performance, but have a distribution with markedly higher density at zero, allowing more parameters to be pruned safely. Powerpropagation is general, intuitive, cheap and straight-forward to implement and can readily be combined with various other techniques. To highlight its versatility, we explore it in two very different settings: Firstly, following a recent line of work, we investigate its effect on sparse training for resource-constrained settings. Here, we combine Powerpropagation with a traditional weight-pruning technique as well as recent state-of-the-art sparse-to-sparse algorithms, showing superior performance on the ImageNet benchmark. Secondly, we advocate the use of sparsity in overcoming catastrophic forgetting, where compressed representations allow accommodating a large number of tasks at fixed model capacity. In all cases our reparameterisation considerably increases the efficacy of the off-the-shelf methods.

Accepted at NeurIPS 2021

References in corpus (35)

• Adam: A Method for Stochastic Optimization
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Overcoming catastrophic forgetting in neural networks
• Language Models are Few-Shot Learners
• Trust Region Policy Optimization
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Reconciling modern machine learning practice and the bias-variance trade-off
• The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
• An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks
• Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles
• PathNet: Evolution Channels Gradient Descent in Super Neural Networks
• Bayesian Online Changepoint Detection
• SNIP: Single-shot Network Pruning based on Connection Sensitivity
• Three scenarios for continual learning
• The State of Sparsity in Deep Neural Networks
• Experience Replay for Continual Learning
• Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning
• Variational Dropout Sparsifies Deep Neural Networks
• Variational Continual Learning
• Re-evaluating Continual Learning Scenarios: A Categorization and Case for Strong Baselines
• Continual Learning with Deep Generative Replay
• Sparse Networks from Scratch: Faster Training without Losing Performance
• Generative replay with feedback connections as a general strategy for continual learning
• Deep Rewiring: Training very sparse deep networks
• Learning Sparse Neural Networks through $L_0$ Regularization
• Stabilizing the Lottery Ticket Hypothesis
• Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterization
• Functional Regularisation for Continual Learning with Gaussian Processes
• SpaceNet: Make Free Space For Continual Learning
• Implicit Regularization for Optimal Sparse Recovery
• The Difficulty of Training Sparse Neural Networks
• Top-KAST: Top-K Always Sparse Training
• Continual World: A Robotic Benchmark For Continual Reinforcement Learning
• Continual Reinforcement Learning with Multi-Timescale Replay
• Supermasks in Superposition