LOss-Based SensiTivity rEgulaRization: towards deep sparse neural networks
arXiv:2011.09905 · doi:10.1016/j.neunet.2021.11.029
Abstract
LOBSTER (LOss-Based SensiTivity rEgulaRization) is a method for training neural networks having a sparse topology. Let the sensitivity of a network parameter be the variation of the loss function with respect to the variation of the parameter. Parameters with low sensitivity, i.e. having little impact on the loss when perturbed, are shrunk and then pruned to sparsify the network. Our method allows to train a network from scratch, i.e. without preliminary learning or rewinding. Experiments on multiple architectures and datasets show competitive compression ratios with minimal computational overhead.
References in corpus (7)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
- Skin Lesion Analysis Toward Melanoma Detection 2018: A Challenge Hosted by the International Skin Imaging Collaboration (ISIC)
- SCOP: Scientific Control for Reliable Neural Network Pruning
- SeReNe: Sensitivity based Regularization of Neurons for Structured Sparsity in Neural Networks
- Pruning artificial neural networks: a way to find well-generalizing, high-entropy sharp minima
- A Statistical Investigation of Long Memory in Language and Music
Cited by in corpus (6)
- Compressing Explicit Voxel Grid Representations: fast NeRFs become also small
- Pruning artificial neural networks: a way to find well-generalizing, high-entropy sharp minima
- HEMP: High-order Entropy Minimization for neural network comPression
- Adaptive Precision Training (AdaPT): A dynamic fixed point quantized training approach for DNNs
- Learn how to Prune Pixels for Multi-view Neural Image-based Synthesis
- Capsule Networks Do Not Need to Model Everything