Training Neural Networks with Fixed Sparse Masks

Abstract

During typical gradient-based training of deep neural networks, all of the model's parameters are updated at each iteration. Recent work has shown that it is possible to update only a small subset of the model's parameters during training, which can alleviate storage and communication requirements. In this paper, we show that it is possible to induce a fixed sparse mask on the model's parameters that selects a subset to update over many iterations. Our method constructs the mask out of the $k$ parameters with the largest Fisher information as a simple approximation as to which parameters are most important for the task at hand. In experiments on parameter-efficient transfer learning and distributed training, we show that our approach matches or exceeds the performance of other methods for training with sparse updates while being more efficient in terms of memory usage and communication costs. We release our code publicly to promote further applications of our approach.

References in corpus (21)

• Overcoming catastrophic forgetting in neural networks
• A Simple Framework for Contrastive Learning of Visual Representations
• Communication-Efficient Learning of Deep Networks from Decentralized Data
• HuggingFace's Transformers: State-of-the-art Natural Language Processing
• Language Models are Few-Shot Learners
• Federated Learning: Strategies for Improving Communication Efficiency
• Federated Optimization: Distributed Machine Learning for On-Device Intelligence
• The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
• HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent
• Learning multiple visual domains with residual adapters
• Big Self-Supervised Models are Strong Semi-Supervised Learners
• New insights and perspectives on the natural gradient method
• Prefix-Tuning: Optimizing Continuous Prompts for Generation
• Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layers
• Learning Sparse Neural Networks through $L_0$ Regularization
• Faster gaze prediction with dense networks and Fisher pruning
• Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
• Where is the Information in a Deep Neural Network?
• Group Fisher Pruning for Practical Network Compression
• Parameter-Efficient Transfer Learning with Diff Pruning
• Simple, Scalable Adaptation for Neural Machine Translation