1.3k citations · 2.2k across the 33 of their papers we have counts for
9 papers · 2 filters
Revisiting "Qualitatively Characterizing Neural Network Optimization Problems"
Jonathan Frankle
We revisit and extend the experiments of Goodfellow et al. (2014), who showed that - for then state-of-the-art networks - "the objective function has a simple, approximately convex…
The Lottery Tickets Hypothesis for Supervised and Self-supervised Pre-training in Computer Vision Models
Tianlong Chen, Jonathan Frankle, Shiyu Chang +4
The computer vision world has been re-gaining enthusiasm in various pre-trained models, including both classical ImageNet supervised pre-training and recently emerged self-supervis…
Pruning Neural Networks at Initialization: Why are We Missing the Mark?
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M. Roy +1
Recent work has explored the possibility of pruning neural networks at initialization. We assess proposals for doing so: SNIP (Lee et al., 2019), GraSP (Wang et al., 2020), SynFlow…
The Lottery Ticket Hypothesis for Pre-trained BERT Networks
Tianlong Chen, Jonathan Frankle, Shiyu Chang +4
In natural language processing (NLP), enormous pre-trained models like BERT have become the standard starting point for training on a range of downstream tasks, and similar trends…
On the Predictability of Pruning Across Scales
Jonathan S. Rosenfeld, Jonathan Frankle, Michael Carbin +1
We show that the error of iteratively magnitude-pruned networks empirically follows a scaling law with interpretable coefficients that depend on the architecture and task. We funct…
What is the State of Neural Network Pruning?
Davis Blalock, Jose Javier Gonzalez Ortiz, Jonathan Frankle +1
Neural network pruning---the task of reducing the size of a network by removing parameters---has been the subject of a great deal of work in recent years. We provide a meta-analysi…
Comparing Rewinding and Fine-tuning in Neural Network Pruning
Alex Renda, Jonathan Frankle, Michael Carbin
Many neural network pruning algorithms proceed in three steps: train the network to completion, remove unwanted structure to compress the network, and retrain the remaining structu…
Training BatchNorm and Only BatchNorm: On the Expressive Power of Random Features in CNNs
Jonathan Frankle, David J. Schwab, Ari S. Morcos
A wide variety of deep learning techniques from style transfer to multitask learning rely on training affine transformations of features. Most prominent among these is the popular…
The Early Phase of Neural Network Training
Jonathan Frankle, David J. Schwab, Ari S. Morcos
Recent studies have shown that many important aspects of neural network learning take place within the very earliest iterations or epochs of training. For example, sparse, trainabl…