4 papers
SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks
Xiangyi Li, Yimin Liu, Wenbo Chen +75
Agent Skills are structured packages of procedural knowledge that augment large language model (LLM) agents at inference time. Despite rapid adoption, there is no standard way to m…
OctoBench: Benchmarking Scaffold-Aware Instruction Following in Repository-Grounded Agentic Coding
Deming Ding, Shichun Liu, Enhui Yang +12
Modern coding scaffolds turn LLMs into capable software agents, but their ability to follow scaffold-specified instructions remains under-examined, especially when constraints are…
Tool-integrated Reinforcement Learning for Repo Deep Search
Zexiong Ma, Chao Peng, Qunhong Zeng +3
Issue localization, the process of identifying code locations that need modification to resolve software issues, is a critical yet challenging task in software development. The sem…
Evaluating Generated Commit Messages with Large Language Models
Qunhong Zeng, Yuxia Zhang, Zexiong Ma +5
Commit messages are essential in software development as they serve to document and explain code changes. Yet, their quality often falls short in practice, with studies showing sig…