Deconstructing the Structure of Sparse Neural Networks

Abstract

Although sparse neural networks have been studied extensively, the focus has been primarily on accuracy. In this work, we focus instead on network structure, and analyze three popular algorithms. We first measure performance when structure persists and weights are reset to a different random initialization, thereby extending experiments in Deconstructing Lottery Tickets (Zhou et al., 2019). This experiment reveals that accuracy can be derived from structure alone. Second, to measure structural robustness we investigate the sensitivity of sparse neural networks to further pruning after training, finding a stark contrast between algorithms. Finally, for a recent dynamic sparsity algorithm we investigate how early in training the structure emerges. We find that even after one epoch the structure is mostly determined, allowing us to propose a more efficient algorithm which does not require dense gradients throughout training. In looking back at algorithms for sparse neural networks and analyzing their performance from a different lens, we uncover several interesting properties and promising directions for future research.

arXiv comment6 pages, 4 figures, Accepted to ML-Retrospectives, Surveys & Meta-Analyses @ NeurIPS 2020 Workshop

References in corpus (10)

• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
• The State of Sparsity in Deep Neural Networks
• Sparse Networks from Scratch: Faster Training without Losing Performance
• Picking Winning Tickets Before Training by Preserving Gradient Flow
• Rigging the Lottery: Making All Tickets Winners
• Training BatchNorm and Only BatchNorm: On the Expressive Power of Random Features in CNNs
• Pruning neural networks without any data by iteratively conserving synaptic flow
• The Difficulty of Training Sparse Neural Networks
• Top-KAST: Top-K Always Sparse Training