12 papers
Can LLMs Test Terminal User Interfaces?
Chao Peng, Ruida Hu, Ajitha Rajan +4
Terminal User Interfaces (TUIs) combine the stateful, screen-oriented behaviour of GUIs with terminal deployment and are now common in developer tools. Yet they lack a dedicated te…
Evaluating Repository-level Software Documentation via Question Answering and Feature-Driven Development
Xinchen Wang, Ruida Hu, Cuiyun Gao +2
Software documentation is crucial for repository comprehension. While Large Language Models (LLMs) advance documentation generation from code snippets to entire repositories, exist…
Evaluating LLM-Based 0-to-1 Software Generation in End-to-End CLI Tool Scenarios
Ruida Hu, Xinchen Wang, Chao Peng +2
The evolution of Large Language Models (LLMs) has catalyzed a paradigm shift towards intent-driven software development, where autonomous agents are expected to design and deliver…
SR-Eval: Evaluating LLMs on Code Generation under Stepwise Requirement Refinement
Zexun Zhan, Shuzheng Gao, Ruida Hu +1
Large language models (LLMs) have achieved remarkable progress in code generation. However, existing benchmarks mainly formalize the task as a static, single-turn problem, overlook…
Benchmarking LLMs for Fine-Grained Code Review with Enriched Context in Practice
Ruida Hu, Xinchen Wang, Xin-Cheng Wen +5
Code review is a cornerstone of software quality assurance, and recent advances in Large Language Models (LLMs) have shown promise in its automation. However, existing benchmarks f…
RepoMasterEval: Evaluating Code Completion via Real-World Repositories
Qinyun Wu, Chao Peng, Pengfei Gao +9
With the growing reliance on automated code completion tools in software development, the need for comprehensive evaluation benchmarks has become critical. Existing benchmarks focu…