743 citations · 803 across the 17 of their papers we have counts for
3 papers · 1 filter
On the special role of class-selective neurons in early training
Omkar Ranadive, Nikhil Thakurdesai, Ari S Morcos +2
It is commonly observed that deep networks trained for classification exhibit class-selective neurons in their early and intermediate layers. Intriguingly, recent studies have show…
Dynamic Masking Rate Schedules for MLM Pretraining
Zachary Ankner, Naomi Saphra, Davis Blalock +2
Most works on transformers trained with the Masked Language Modeling (MLM) objective use the original BERT model's fixed masking rate of 15%. We propose to instead dynamically sche…
Knowledge Distillation for Efficient Sequences of Training Runs
Xingyu Liu, Alex Leonardi, Lu Yu +3
In many practical scenarios -- like hyperparameter search or continual retraining with new data -- related training runs are performed many times in sequence. Current practice is t…