50 citations · 93 across the 6 of their papers we have counts for
5 papers · 1 filter
Cliff-Learning
Tony T. Wang, Igor Zablotchi, Nir Shavit +1
We study the data-scaling of transfer learning from foundation models in the low-downstream-data regime. We observe an intriguing phenomenon which we call cliff-learning. Cliff-lea…
Scaling Laws for Deep Learning
Jonathan S. Rosenfeld
Running faster will only get you so far -- it is generally advisable to first understand where the roads lead, then get a car ... The renaissance of machine learning (ML) and deep…
On the Predictability of Pruning Across Scales
Jonathan S. Rosenfeld, Jonathan Frankle, Michael Carbin +1
We show that the error of iteratively magnitude-pruned networks empirically follows a scaling law with interpretable coefficients that depend on the architecture and task. We funct…
Self-Play Learning Without a Reward Metric
Dan Schmidt, Nick Moran, Jonathan S. Rosenfeld +2
The AlphaZero algorithm for the learning of strategy games via self-play, which has produced superhuman ability in the games of Go, chess, and shogi, uses a quantitative reward fun…
A Constructive Prediction of the Generalization Error Across Scales
Jonathan S. Rosenfeld, Amir Rosenfeld, Yonatan Belinkov +1
The dependency of the generalization error of neural networks on model and dataset size is of critical importance both in practice and for understanding the theory of neural networ…