5 citations · 5 across the 6 of their papers we have counts for
4 papers · 1 filter
Small LLMs: Pruning vs. Training from Scratch
Yufeng Xu, Taiming Lu, Kunjun Li +3
Pruning promises a shortcut to strong small language models. In this work, we examine this promise by pruning Llama-3.1-8B at pruning ratios of 0.5--0.8 with six methods spanning d…
Strong Teacher Not Needed? On Distillation in LLM Pretraining
Taiming Lu, Zhuang Liu
Knowledge distillation generally assumes a strong-to-weak relationship where stronger teachers yield better students. In this work, we examine this assumption about distillation in…
FrontierCS: Evolving Challenges for Evolving Intelligence
Qiuyang Mang, Wenhao Chai, Zhifei Li +48
We introduce FrontierCS, a benchmark of 156 open-ended problems across diverse areas of computer science, designed and reviewed by experts, including CS PhDs and top-tier competiti…
Transformers without Normalization
Jiachen Zhu, Xinlei Chen, Kaiming He +2
Normalization layers are ubiquitous in modern neural networks and have long been considered essential. This work demonstrates that Transformers without normalization can achieve th…