4 papers
What Process Evaluation of Coding Agents Actually Measures: Action, Task, and Step Are Three Different Levels
Jiawei He, Mengyu Shi, Jie jia +2
Coding agents are increasingly evaluated not only by whether they solve a task, but also by how they execute it. However, existing process-level evaluations often treat action pred…
From Fragments to Paths: Task-Level Context Recovery for Large Industrial Codebases
Jiawei He, Weisong Sun, Mengyu Shi +4
Large language models have shown strong performance on software engineering (SE) tasks, yet understanding large industrial repositories remains challenging. Existing methods often…
SSDAU: Structured Semantic Data Augmentation for Joint Entity and Relation Extraction
Jiawei He, Mengyu Shi, Jiawei Liu +6
Joint Entity and Relation Extraction (JERE) is highly sensitive to training data quality, making data augmentation a natural way to improve generalization. However, existing augmen…
ProcCtrlBench: Evaluating Process-Level Defects and Control Preservation in LLM Coding Agents
Jiawei He, Jie Jia, Chenbo Liu +4
Existing benchmarks for LLM coding agents primarily evaluate final outcomes. While useful for measuring overall capability, these metrics provide limited visibility and often miss…