11 papers
COMPAS: Difficulty-Aware Joint Search for Optimizing Code Generation
Jingzhi Gong, Jie M. Zhang, Gunel Jahangirova +3
Code generation systems make each LLM call with a model, a prompt, and decoding settings. However, existing optimization methods usually tune only part of these choices or use one…
A Study of LLMs' Preferences for Libraries and Programming Languages
Lukas Twist, Mark Harman, Don Syme +4
Despite the rapid progress of large language models (LLMs) in code generation, existing evaluations focus on functional correctness or syntactic validity, overlooking how LLMs make…
TerminalWorld: Benchmarking Agents on Real-World Terminal Tasks
Zhaoyang Chu, Jiarui Hu, Xingyu Jiang +8
We introduce TerminalWorld, a scalable data engine that automatically reverse-engineers high-fidelity evaluation tasks from "in-the-wild" terminal recordings. Processing 80,870 ter…
Library Hallucinations in LLM-Generated Code: A Risk Analysis Grounded in Developer Queries
Lukas Twist, Jie M. Zhang, Mark Harman +1
Large language models (LLMs) now play a central role in code generation, yet they continue to hallucinate, frequently inventing non-existent libraries. Such library hallucinations…
EET: Experience-Driven Early Termination for Cost-Efficient Software Engineering Agents
Yaoqi Guo, Ying Xiao, Jie M. Zhang +4
Software engineering (SE) agents powered by large language models are increasingly adopted in practice, yet they often incur substantial monetary cost. We introduce EET, an experie…
A Comprehensive Study on Large Language Models for Mutation Testing
Bo Wang, Mingda Chen, Ming Deng +4
Large Language Models (LLMs) have recently been used to generate mutants in both research work and in industrial practice. However, there has been no comprehensive empirical study…