24 papers
RealBench: A Repo-Level Code Generation Benchmark Aligned with Real-World Software Development Practices
Jia Li, Hongyi Deng, Yiran Zhang +9
Writing code requires significant time and effort in software development. To automate this process, researchers have made substantial progress using Large Language Models (LLMs) f…
ClarifyCodeBench: Evaluating LLMs on Clarifying Ambiguous Requirements for Code Generation
Zheng Fang, Dongming Jin, Yihong dong +4
Large Language Models have emerged as programming assistants. However, the efficacy of code generation is constrained by the quality of input requirements, which are frequently amb…
Weights to Code: Extracting Interpretable Algorithms from the Discrete Transformer
Yifan Zhang, Wei Bi, Kechi Zhang +3
Algorithm extraction aims to synthesize executable programs directly from models trained on algorithmic tasks, enabling de novo recovery of executable mechanisms from weights witho…
KOCO-BENCH: Can Large Language Models Leverage Domain Knowledge in Software Development?
Xue Jiang, Ge Li, Jiaru Qian +12
Large language models (LLMs) excel at general programming but struggle with domain-specific software development, necessitating domain specialization methods for LLMs to learn and…
Evaluating the Formal Reasoning Capabilities of Large Language Models through Chomsky Hierarchy
Yihong Dong, Jianha Xiao, Xue Jiang +7
The formal reasoning capabilities of LLMs are crucial for advancing automated software engineering. However, existing benchmarks for LLMs lack systematic evaluation based on comput…
RL-PLUS: Countering Capability Boundary Collapse of LLMs in Reinforcement Learning with Hybrid-policy Optimization
Yihong Dong, Xue Jiang, Yongding Tao +11
Reinforcement Learning with Verifiable Reward (RLVR) has significantly advanced the complex reasoning abilities of Large Language Models (LLMs). However, it struggles to break thro…