6 papers
Grounded Chess Reasoning in Language Models via Master Distillation
Zhenwei Tang, Qianfeng Wen, Seth Grief-Albert +4
Language models often lack grounded reasoning capabilities in specialized domains where training data is scarce but bespoke systems excel. We introduce a general framework for dist…
Precise Debugging Benchmark: Is Your Model Debugging or Regenerating?
Wang Bill Zhu, Miaosen Chai, Shangshang Wang +5
Unlike code completion, debugging requires localizing faults and applying targeted edits. We observe that frontier LLMs often regenerate correct but over-edited solutions during de…
SWE-Bench Mobile: Can Large Language Model Agents Develop Industry-Level Mobile Applications?
Muxin Tian, Zhe Wang, Blair Yang +7
Can large language model agents develop industry-level mobile applications? We introduce \textbf{SWE-Bench Mobile}, a benchmark for evaluating coding agents on realistic software e…
LM Agents May Fail to Act on Their Own Risk Knowledge
Yuzhi Tang, Tianxiao Li, Elizabeth Li +3
Language model (LM) agents have demonstrated significant potential for automating real-world tasks, yet they pose a diverse array of potential, severe risks in safety-critical scen…
TypyBench: Evaluating LLM Type Inference for Untyped Python Repositories
Honghua Dong, Jiacheng Yang, Xun Deng +4
Type inference for dynamic languages like Python is a persistent challenge in software engineering. While large language models (LLMs) have shown promise in code understanding, the…
-Bench: Evaluating Conversational Agents in a Dual-Control Environment
Victor Barres, Honghua Dong, Soham Ray +2
Existing benchmarks for conversational AI agents simulate single-control environments, where only the AI agent can use tools to interact with the world, while the user remains a pa…