4 papers
Precise Debugging Benchmark: Is Your Model Debugging or Regenerating?
Wang Bill Zhu, Miaosen Chai, Shangshang Wang +5
Unlike code completion, debugging requires localizing faults and applying targeted edits. We observe that frontier LLMs often regenerate correct but over-edited solutions during de…
Generating Complex Code Analyzers from Natural Language Questions
Amirmohammad Nazari, Sadra Sabouri, Wang Bill Zhu +3
Many software development tasks, such as implementing features and fixing bugs, begin with developers posing questions about a codebase. However, answering questions about codebase…
PDDL-Mind: Large Language Models are Capable on Belief Reasoning with Reliable State Tracking
Wang Bill Zhu, Qiutong Tony Yi, Robin Jia +1
Large language models (LLMs) perform substantially below human level on existing theory-of-mind (ToM) benchmarks, even when augmented with chain-of-thought prompting or probabilist…
Self-Evolving LLM Memory Extraction Across Heterogeneous Tasks
Yuqing Yang, Tengxiao Liu, Wang Bill Zhu +3
As LLM-based assistants become persistent and personalized, they must extract and retain useful information from past conversations as memory. However, the types of information wor…