Showing cs.LGShow all
3 papers · 1 filter
cs.LG2026
Small LLMs: Pruning vs. Training from Scratch
Yufeng Xu, Taiming Lu, Kunjun Li +3
Pruning promises a shortcut to strong small language models. In this work, we examine this promise by pruning Llama-3.1-8B at pruning ratios of 0.5--0.8 with six methods spanning d…
cs.LG2026
Strong Teacher Not Needed? On Distillation in LLM Pretraining
Taiming Lu, Zhuang Liu
Knowledge distillation generally assumes a strong-to-weak relationship where stronger teachers yield better students. In this work, we examine this assumption about distillation in…
cs.LG2026
Stronger Normalization-Free Transformers
Mingzhi Chen, Taiming Lu, Jiachen Zhu +2
Although normalization layers have long been viewed as indispensable components of deep learning architectures, the recent introduction of Dynamic Tanh (DyT) has demonstrated that…