5 papers · 1 filter
Beyond Task Completion: Measuring Interaction Cost in Terminal User Interfaces
Ruida Hu, Yuanhao Wang, Chao Peng +2
Large language models (LLMs) are increasingly used through terminal user interfaces (TUIs), yet task completion alone does not capture how difficult an interface is to understand a…
Can LLMs Test Terminal User Interfaces?
Chao Peng, Ruida Hu, Ajitha Rajan +3
Terminal User Interfaces (TUIs) combine the stateful, screen-oriented behaviour of GUIs with terminal deployment and are now common in developer tools. Yet they lack a dedicated te…
TerminalWorld: Benchmarking Agents on Real-World Terminal Tasks
Zhaoyang Chu, Jiarui Hu, Xingyu Jiang +8
We introduce TerminalWorld, a scalable data engine that automatically reverse-engineers high-fidelity evaluation tasks from "in-the-wild" terminal recordings. Processing 80,870 ter…
Evaluating Repository-level Software Documentation via Question Answering and Feature-Driven Development
Xinchen Wang, Ruida Hu, Cuiyun Gao +2
Software documentation is crucial for repository comprehension. While Large Language Models (LLMs) advance documentation generation from code snippets to entire repositories, exist…
Evaluating LLM-Based 0-to-1 Software Generation in End-to-End CLI Tool Scenarios
Ruida Hu, Xinchen Wang, Chao Peng +2
The evolution of Large Language Models (LLMs) has catalyzed a paradigm shift towards intent-driven software development, where autonomous agents are expected to design and deliver…