5 papers
ICAE-Bench: Evaluating Coding Agents as Interactive Project Builders
Zhongyuan Peng, Dan Huang, Chuyu Zhang +8
The recent emergence of vibe-coding workflows is changing what coding agents are expected to do. Instead of merely completing code under fully specified instructions, agents are in…
What Is Your Agent's GPA? A Framework for Evaluating Agent Goal-Plan-Action Alignment
Allison Sihan Jia, Daniel Huang, Nikhil Vytla +5
We introduce the Agent GPA (Goal-Plan-Action) framework, driven by the fundamental insight that critical agent failures emerge at the intersections of setting goals, devising plans…
A Computational Approach to Analyzing Language Change and Variation in the Constructed Language Toki Pona
Daniel Huang, Hyoun-A Joo
This study explores language change and variation in Toki Pona, a constructed language with approximately 120 core words. Taking a computational and corpus-based approach, the stud…
Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarks
Yixin Cao, Shibo Hong, Xinze Li +24
Large Language Models (LLMs) are advancing at an amazing speed and have become indispensable across academia, industry, and daily applications. To keep pace with the status quo, th…
LessLeak-Bench: A First Investigation of Data Leakage in LLMs Across 83 Software Engineering Benchmarks
Xin Zhou, Martin Weyssow, Ratnadira Widyasari +7
Large Language Models (LLMs) are widely utilized in software engineering (SE) tasks, such as code generation and automated program repair. However, their reliance on extensive and…