672 citations · 1.3k across the 6 of their papers we have counts for
Showing 2022Show all
2 papers · 1 filter
cs.CL2022★ 672 cited
Training Compute-Optimal Large Language Models
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch +19
We investigate the optimal model size and number of tokens for training a transformer language model under a given compute budget. We find that current large language models are si…
cs.CL2022★ 23 cited
Unified Scaling Laws for Routed Language Models
Aidan Clark, Diego de las Casas, Aurelia Guy +23
The performance of a language model has been shown to be effectively modeled as a power-law in its parameter count. Here we study the scaling behaviors of Routing Networks: archite…