6 papers
Scaling Laws for Code: A More Data-Hungry Regime
Xianzhen Luo, Wenzhen Zheng, Qingfu Zhu +5
Code Large Language Models (LLMs) are revolutionizing software engineering. However, scaling laws that guide the efficient training are predominantly analyzed on Natural Language (…
How Many Code and Test Cases Are Enough? Evaluating Test Cases Generation from a Binary-Matrix Perspective
Xianzhen Luo, Jinyang Huang, Wenzhen Zheng +5
Evaluating test cases automatically generated by Large Language Models (LLMs) is a critical yet challenging task. Existing benchmarks often evaluate the exclusion ratio on large, u…
Predictable Scale: Part I, Step Law -- Optimal Hyperparameter Scaling Law in Large Language Model Pretraining
Houyi Li, Wenzhen Zheng, Qiufeng Wang +10
The impressive capabilities of Large Language Models (LLMs) across diverse tasks are now well established, yet their effective deployment necessitates careful hyperparameter optimi…
Step-3 is Large yet Affordable: Model-system Co-design for Cost-effective Decoding
StepFun, :, Bin Wang +195
Large language models (LLMs) face low hardware efficiency during decoding, especially for long-context reasoning tasks. This paper introduces Step-3, a 321B-parameter VLM with hard…
Success is in the Details: Evaluate and Enhance Details Sensitivity of Code LLMs through Counterfactuals
Xianzhen Luo, Qingfu Zhu, Zhiming Zhang +8
Code Sensitivity refers to the ability of Code LLMs to recognize and respond to details changes in problem descriptions. While current code benchmarks and instruction data focus on…
OpenCoder: The Open Cookbook for Top-Tier Code Large Language Models
Siming Huang, Tianhao Cheng, J. K. Liu +16
Large language models (LLMs) for code have become indispensable in various domains, including code generation, reasoning tasks and agent systems. While open-access code LLMs are in…