Small LLMs: Pruning vs. Training from Scratch

Yufeng Xu, Taiming Lu, Kunjun Li +3

Pruning promises a shortcut to strong small language models. In this work, we examine this promise by pruning Llama-3.1-8B at pruning ratios of 0.5--0.8 with six methods spanning d…