2 citations · 3 across the 5 of their papers we have counts for
5 papers
Predictable Scale: Part II, Farseer: A Refined Scaling Law in Large Language Models
Houyi Li, Wenzhen Zheng, Qiufeng Wang +8
Training Large Language Models (LLMs) is prohibitively expensive, creating a critical scaling gap where insights from small-scale experiments often fail to transfer to resource-int…
Mixture-of-Experts Can Surpass Dense LLMs Under Strictly Equal Resource
Houyi Li, Ka Man Lo, Shijie Xuyang +7
Mixture-of-Experts (MoE) language models dramatically expand model capacity and achieve remarkable performance without increasing per-token compute. However, can MoEs surpass dense…
Success is in the Details: Evaluate and Enhance Details Sensitivity of Code LLMs through Counterfactuals
Xianzhen Luo, Qingfu Zhu, Zhiming Zhang +8
Code Sensitivity refers to the ability of Code LLMs to recognize and respond to details changes in problem descriptions. While current code benchmarks and instruction data focus on…
Is Compression Really Linear with Code Intelligence?
Shijie Xuyang, Xianzhen Luo, Zheng Chu +6
Understanding the relationship between data compression and the capabilities of Large Language Models (LLMs) is crucial, especially in specialized domains like code intelligence. P…
Predictable Scale: Part I, Step Law -- Optimal Hyperparameter Scaling Law in Large Language Model Pretraining
Houyi Li, Wenzhen Zheng, Qiufeng Wang +10
The impressive capabilities of Large Language Models (LLMs) across diverse tasks are now well established, yet their effective deployment necessitates careful hyperparameter optimi…