Neural Parameter Allocation Search
arXiv:2006.10598
Abstract
Training neural networks requires increasing amounts of memory. Parameter sharing can reduce memory and communication costs, but existing methods assume networks have many identical layers and utilize hand-crafted sharing strategies that fail to generalize. We introduce Neural Parameter Allocation Search (NPAS), a novel task where the goal is to train a neural network given an arbitrary, fixed parameter budget. NPAS covers both low-budget regimes, which produce compact networks, as well as a novel high-budget regime, where additional capacity can be added to boost performance without increasing inference FLOPs. To address NPAS, we introduce Shapeshifter Networks (SSNs), which automatically learn where and how to share parameters in a network to support any parameter budget without requiring any changes to the architecture or loss function. NPAS and SSNs provide a complete framework for addressing generalized parameter sharing, and can also be combined with prior work for additional performance gains. We demonstrate the effectiveness of our approach using nine network architectures across four diverse tasks, including ImageNet classification and transformers.
arXiv commentAccepted at ICLR 2022
References in corpus (17)
- Distilling the Knowledge in a Neural Network
- Neural Architecture Search with Reinforcement Learning
- Federated Learning: Strategies for Improving Communication Efficiency
- Improved Regularization of Convolutional Neural Networks with Cutout
- Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer
- Binarized Neural Networks
- Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
- SNIP: Single-shot Network Pruning based on Connection Sensitivity
- Deep Equilibrium Models
- One-Shot Neural Architecture Search via Self-Evaluated Template Network
- The Reversible Residual Network: Backpropagation Without Storing Activations
- Mixed Precision Training of Convolutional Neural Networks using Integer Operations
- BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learning
- Knowledge Transfer with Jacobian Matching
- Neural Relational Inference with Fast Modular Meta-learning
- Learning Implicitly Recurrent CNNs Through Parameter Sharing
- ProdSumNet: reducing model parameters in deep neural networks via product-of-sums matrix decompositions