1.3k citations · 2.2k across the 33 of their papers we have counts for
6 papers · 2 filters
The Effect of Data Dimensionality on Neural Network Prunability
Zachary Ankner, Alex Renda, Gintare Karolina Dziugaite +2
Practitioners prune neural networks for efficiency gains and generalization improvements, but few scrutinize the factors determining the prunability of a neural network the maximum…
Reduce, Reuse, Recycle: Improving Training Efficiency with Distillation
Cody Blakeney, Jessica Zosa Forde, Jonathan Frankle +2
Methods for improving the efficiency of deep network training (i.e. the resources required to achieve a given level of model quality) are of immediate benefit to deep learning prac…
Pruning's Effect on Generalization Through the Lens of Training and Regularization
Tian Jin, Michael Carbin, Daniel M. Roy +2
Practitioners frequently observe that pruning improves model generalization. A long-standing hypothesis based on bias-variance trade-off attributes this generalization improvement…
Unmasking the Lottery Ticket Hypothesis: What's Encoded in a Winning Ticket's Mask?
Mansheej Paul, Feng Chen, Brett W. Larsen +3
Modern deep learning involves training costly, highly overparameterized networks, thus motivating the search for sparser networks that can still be trained to the same accuracy as…
Lottery Tickets on a Data Diet: Finding Initializations with Sparse Trainable Networks
Mansheej Paul, Brett W. Larsen, Surya Ganguli +2
A striking observation about iterative magnitude pruning (IMP; Frankle et al. 2020) is that $\unicode{x2014}$ after just a few hundred steps of dense training $\unicode{x2014}$ the…
Fast Benchmarking of Accuracy vs. Training Time with Cyclic Learning Rates
Jacob Portes, Davis Blalock, Cory Stephenson +1
Benchmarking the tradeoff between neural network accuracy and training time is computationally expensive. Here we show how a multiplicative cyclic learning rate schedule can be use…