9 papers
Beyond Task Completion: Measuring Interaction Cost in Terminal User Interfaces
Ruida Hu, Yuanhao Wang, Chao Peng +2
Large language models (LLMs) are increasingly used through terminal user interfaces (TUIs), yet task completion alone does not capture how difficult an interface is to understand a…
Can LLMs Test Terminal User Interfaces?
Chao Peng, Ruida Hu, Ajitha Rajan +3
Terminal User Interfaces (TUIs) combine the stateful, screen-oriented behaviour of GUIs with terminal deployment and are now common in developer tools. Yet they lack a dedicated te…
TerminalWorld: Benchmarking Agents on Real-World Terminal Tasks
Zhaoyang Chu, Jiarui Hu, Xingyu Jiang +8
We introduce TerminalWorld, a scalable data engine that automatically reverse-engineers high-fidelity evaluation tasks from "in-the-wild" terminal recordings. Processing 80,870 ter…
Evaluating Repository-level Software Documentation via Question Answering and Feature-Driven Development
Xinchen Wang, Ruida Hu, Cuiyun Gao +2
Software documentation is crucial for repository comprehension. While Large Language Models (LLMs) advance documentation generation from code snippets to entire repositories, exist…
Evaluating LLM-Based 0-to-1 Software Generation in End-to-End CLI Tool Scenarios
Ruida Hu, Xinchen Wang, Chao Peng +2
The evolution of Large Language Models (LLMs) has catalyzed a paradigm shift towards intent-driven software development, where autonomous agents are expected to design and deliver…
Benchmarking LLMs for Fine-Grained Code Review with Enriched Context in Practice
Ruida Hu, Xinchen Wang, Xin-Cheng Wen +5
Code review is a cornerstone of software quality assurance, and recent advances in Large Language Models (LLMs) have shown promise in its automation. However, existing benchmarks f…
CodeRepoQA: A Large-scale Benchmark for Software Engineering Question Answering
Ruida Hu, Chao Peng, Jingyi Ren +6
In this work, we introduce CodeRepoQA, a large-scale benchmark specifically designed for evaluating repository-level question-answering capabilities in the field of software engine…
DialogAgent: An Auto-engagement Agent for Code Question Answering Data Production
Xiaoyun Liang, Jingyi Ren, Jiayi Qi +2
Large Language Models (LLMs) have become increasingly integral to enhancing developer productivity, particularly in code generation, comprehension, and repair tasks. However, fine-…
ContextModule: Improving Code Completion via Repository-level Contextual Information
Zhanming Guan, Junlin Liu, Jierui Liu +7
Large Language Models (LLMs) have demonstrated impressive capabilities in code completion tasks, where they assist developers by predicting and generating new code in real-time. Ho…