LNPT: Label-free Network Pruning and Training
arXiv:2403.12690 · doi:10.1109/IJCNN60899.2024.10651544
Abstract
Pruning before training enables the deployment of neural networks on smart devices. By retaining weights conducive to generalization, pruned networks can be accommodated on resource-constrained smart devices. It is commonly held that the distance on weight norms between the initialized and the fully-trained networks correlates with generalization performance. However, as we have uncovered, inconsistency between this metric and generalization during training processes, which poses an obstacle to determine the pruned structures on smart devices in advance. In this paper, we introduce the concept of the learning gap, emphasizing its accurate correlation with generalization. Experiments show that the learning gap, in the form of feature maps from the penultimate layer of networks, aligns with variations of generalization performance. We propose a novel learning framework, LNPT, which enables mature networks on the cloud to provide online guidance for network pruning and learning on smart devices with unlabeled data. Our results demonstrate the superiority of this approach over supervised training.
IJCNN 2024
References in corpus (14)
- Distilling the Knowledge in a Neural Network
- Decoupled Weight Decay Regularization
- FitNets: Hints for Thin Deep Nets
- Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer
- Neural Tangent Kernel: Convergence and Generalization in Neural Networks
- The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
- SNIP: Single-shot Network Pruning based on Connection Sensitivity
- Like What You Like: Knowledge Distill via Neuron Selectivity Transfer
- Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent
- Generalization in Deep Networks: The Role of Distance from Initialization
- Drawing Early-Bird Tickets: Towards More Efficient Training of Deep Networks
- Model Pruning Based on Quantified Similarity of Feature Maps
- SEVEN: Pruning Transformer Model by Reserving Sentinels
- Receding Neuron Importances for Structured Pruning