1.3k citations · 2.2k across the 33 of their papers we have counts for
11 papers · 1 filter
Revisiting "Qualitatively Characterizing Neural Network Optimization Problems"
Jonathan Frankle
We revisit and extend the experiments of Goodfellow et al. (2014), who showed that - for then state-of-the-art networks - "the objective function has a simple, approximately convex…
The Lottery Tickets Hypothesis for Supervised and Self-supervised Pre-training in Computer Vision Models
Tianlong Chen, Jonathan Frankle, Shiyu Chang +4
The computer vision world has been re-gaining enthusiasm in various pre-trained models, including both classical ImageNet supervised pre-training and recently emerged self-supervis…
Are all negatives created equal in contrastive instance discrimination?
Tiffany Tianhui Cai, Jonathan Frankle, David J. Schwab +1
Self-supervised learning has recently begun to rival supervised learning on computer vision tasks. Many of the recent approaches have been based on contrastive instance discriminat…
Pruning Neural Networks at Initialization: Why are We Missing the Mark?
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M. Roy +1
Recent work has explored the possibility of pruning neural networks at initialization. We assess proposals for doing so: SNIP (Lee et al., 2019), GraSP (Wang et al., 2020), SynFlow…
The Lottery Ticket Hypothesis for Pre-trained BERT Networks
Tianlong Chen, Jonathan Frankle, Shiyu Chang +4
In natural language processing (NLP), enormous pre-trained models like BERT have become the standard starting point for training on a range of downstream tasks, and similar trends…
On the Predictability of Pruning Across Scales
Jonathan S. Rosenfeld, Jonathan Frankle, Michael Carbin +1
We show that the error of iteratively magnitude-pruned networks empirically follows a scaling law with interpretable coefficients that depend on the architecture and task. We funct…