99 citations · 115 across the 13 of their papers we have counts for
Showing 2024Show all
2 papers · 1 filter
cs.CL2024
Gradient Localization Improves Lifelong Pretraining of Language Models
Jared Fernandez, Yonatan Bisk, Emma Strubell
Large Language Models (LLMs) trained on web-scale text corpora have been shown to capture world knowledge in their parameters. However, the mechanism by which language models store…
cs.LG2024★ 2 cited
Hardware Scaling Trends and Diminishing Returns in Large-Scale Distributed Training
Jared Fernandez, Luca Wehrstedt, Leonid Shamis +5
Dramatic increases in the capabilities of neural network models in recent years are driven by scaling model size, training data, and corresponding computational resources. To devel…