Performance of Small Language Model Pretraining on FABRIC: An Empirical Study

Praveen Rao

Large language models (LLMs) require enormous computing power to pretrain on massive datasets. When limited datasets are available, smaller-sized LLMs are better choice to pretrain…