Where to Begin: Efficient Pretraining via Subnetwork Selection and Distillation

Arjun Krishnakumar, Rhea Sanjay Sukthanker, Hannan Javed Mahadik +5

Small Language models (SLMs) offer an efficient and accessible alternative to Large Language Models (LLMs), delivering strong performance while using far fewer resources. We introd…