Learning Sparse Networks Using Targeted Dropout

Abstract

Neural networks are easier to optimise when they have many more weights than are required for modelling the mapping from inputs to outputs. This suggests a two-stage learning procedure that first learns a large net and then prunes away connections or hidden units. But standard training does not necessarily encourage nets to be amenable to pruning. We introduce targeted dropout, a method for training a neural network so that it is robust to subsequent pruning. Before computing the gradients for each weight update, targeted dropout stochastically selects a set of units or weights to be dropped using a simple self-reinforcing sparsity criterion and then computes the gradients for the remaining weights. The resulting network is robust to post hoc pruning of weights or units that frequently occur in the dropped sets. The method improves upon more complicated sparsifying regularisers while being simple to implement and easy to tune.

References in corpus (19)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Improving neural networks by preventing co-adaptation of feature detectors
• WaveNet: A Generative Model for Raw Audio
• Wide Residual Networks
• Pruning Convolutional Neural Networks for Resource Efficient Inference
• The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
• Rethinking the Value of Network Pruning
• Variational Dropout and the Local Reparameterization Trick
• The State of Sparsity in Deep Neural Networks
• Neural Machine Translation in Linear Time
• Flipout: Efficient Pseudo-Independent Weight Perturbations on Mini-Batches
• Learning Sparse Neural Networks through $L_0$ Regularization
• Faster gaze prediction with dense networks and Fisher pruning
• meProp: Sparsified Back Propagation for Accelerated Deep Learning with Reduced Overfitting
• Structured Probabilistic Pruning for Convolutional Neural Network Acceleration
• Learning Ordered Representations with Nested Dropout
• Learning to Prune Filters in Convolutional Neural Networks
• NoiseOut: A Simple Way to Prune Neural Networks
• Smallify: Learning Network Size while Training

Cited by in corpus (22)

• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• Discovering Neural Wirings
• LOss-Based SensiTivity rEgulaRization: towards deep sparse neural networks
• CARS: Continuous Evolution for Efficient Neural Architecture Search
• SeReNe: Sensitivity based Regularization of Neurons for Structured Sparsity in Neural Networks
• Semi-Supervised Deep Learning for Multi-Tissue Segmentation from Multi-Contrast MRI
• Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Training
• DeepHist: Differentiable Joint and Color Histogram Layers for Image-to-Image Translation
• On Convergence and Generalization of Dropout Training
• Pruning via Iterative Ranking of Sensitivity Statistics
• Convolutional Neural Networks for Speech Controlled Prosthetic Hands
• Lifelong Learning of Compositional Structures
• Boosting Co-teaching with Compression Regularization for Label Noise
• Weight Pruning via Adaptive Sparsity Loss
• Consistent Sparse Deep Learning: Theory and Computation
• Reinforcement Learning with Chromatic Networks for Compact Architecture Search
• Learning by Active Forgetting for Neural Networks
• The Role of Regularization in Shaping Weight and Node Pruning Dependency and Dynamics
• Explore the Knowledge contained in Network Weights to Obtain Sparse Neural Networks
• Back to Square One: Superhuman Performance in Chutes and Ladders Through Deep Neural Networks and Tree Search
• The Impact of Activation Sparsity on Overfitting in Convolutional Neural Networks
• Out-of-the-box channel pruned networks