LOss-Based SensiTivity rEgulaRization: towards deep sparse neural networks
arXiv:2011.09905 · doi:10.1016/j.neunet.2021.11.029
Abstract
LOBSTER (LOss-Based SensiTivity rEgulaRization) is a method for training neural networks having a sparse topology. Let the sensitivity of a network parameter be the variation of the loss function with respect to the variation of the parameter. Parameters with low sensitivity, i.e. having little impact on the loss when perturbed, are shrunk and then pruned to sparsify the network. Our method allows to train a network from scratch, i.e. without preliminary learning or rewinding. Experiments on multiple architectures and datasets show competitive compression ratios with minimal computational overhead.
References in corpus (12)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
- The HAM10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions
- The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
- Skin Lesion Analysis Toward Melanoma Detection 2018: A Challenge Hosted by the International Skin Imaging Collaboration (ISIC)
- SNIP: Single-shot Network Pruning based on Connection Sensitivity
- Variational Dropout Sparsifies Deep Neural Networks
- SCOP: Scientific Control for Reliable Neural Network Pruning
- Learning Sparse Networks Using Targeted Dropout
- SeReNe: Sensitivity based Regularization of Neurons for Structured Sparsity in Neural Networks
- Pruning artificial neural networks: a way to find well-generalizing, high-entropy sharp minima
- A Statistical Investigation of Long Memory in Language and Music
Cited by in corpus (6)
- Compressing Explicit Voxel Grid Representations: fast NeRFs become also small
- Pruning artificial neural networks: a way to find well-generalizing, high-entropy sharp minima
- HEMP: High-order Entropy Minimization for neural network comPression
- Adaptive Precision Training (AdaPT): A dynamic fixed point quantized training approach for DNNs
- Learn how to Prune Pixels for Multi-view Neural Image-based Synthesis
- Capsule Networks Do Not Need to Model Everything