8 papers
LangChoiceBench: Measuring and Explaining Programming-Language Choice in LLMs
Lukas Twist, Twm Stone, Helen Yannakoudakis +1
Large language models (LLMs) have been shown to exhibit strong Python preferences when generating project-level code, but there is currently no systematic way to measure this behav…
A Study of LLMs' Preferences for Libraries and Programming Languages
Lukas Twist, Mark Harman, Don Syme +4
Despite the rapid progress of large language models (LLMs) in code generation, existing evaluations focus on functional correctness or syntactic validity, overlooking how LLMs make…
Reasoning-Trace Collapse: Evaluating the Loss of Explicit Reasoning During Fine-Tuning
Lukas Twist, Helen Yannakoudakis, Jie M. Zhang
Explicit reasoning models are trained to produce intermediate reasoning traces before final answers, but downstream fine-tuning is often performed on ordinary instruction-response…
Library Hallucinations in LLM-Generated Code: A Risk Analysis Grounded in Developer Queries
Lukas Twist, Jie M. Zhang, Mark Harman +1
Large language models (LLMs) now play a central role in code generation, yet they continue to hallucinate, frequently inventing non-existent libraries. Such library hallucinations…
SkillMOO: Multi-Objective Optimization of Agent Skills for Software Engineering
Jingzhi Gong, Ruizhen Gu, Zhiwei Fei +7
Agent skills are increasingly used to configure coding agents for software engineering (SE) tasks, yet current practice treats them as static, hand-crafted assets, or evolved on pa…
Not All Code Is Equal: A Data-Centric Study of Code Complexity and LLM Reasoning
Lukas Twist, Shu Yang, Hanqi Yan +4
Large Language Models (LLMs) increasingly exhibit strong reasoning abilities, often attributed to their capacity to generate chain-of-thought-style intermediate reasoning. Recent w…