5 papers · 1 filter
OccuBench: Evaluating AI Agents on Real-World Professional Tasks via Language Environment Simulation
Xiaomeng Hu, Yinger Zhang, Fei Huang +7
AI agents are expected to perform professional work across hundreds of occupational domains (from emergency department triage to nuclear reactor safety monitoring to customs import…
Format-Adapter: Improving Reasoning Capability of LLMs by Adapting Suitable Format
Dingzirui Wang, Xuanliang Zhang, Rongyu Cao +8
Generating and voting multiple answers is an effective method to mitigate reasoning inconsistencies of large language models (LLMs). Prior works have shown that multiple reasoning…
In-Context Transfer Learning: Demonstration Synthesis by Transferring Similar Tasks
Dingzirui Wang, Xuanliang Zhang, Qiguang Chen +9
In-context learning (ICL) is an effective approach to help large language models (LLMs) adapt to various tasks by providing demonstrations of the target task. Considering the high…
EvoCodeBench: An Evolving Code Generation Benchmark with Domain-Specific Evaluations
Jia Li, Ge Li, Xuanming Zhang +6
How to evaluate Large Language Models (LLMs) in code generation remains an open question. Existing benchmarks have two limitations - data leakage and lack of domain-specific evalua…
DevEval: A Manually-Annotated Code Generation Benchmark Aligned with Real-World Code Repositories
Jia Li, Ge Li, Yunfei Zhao +15
How to evaluate the coding abilities of Large Language Models (LLMs) remains an open question. We find that existing benchmarks are poorly aligned with real-world code repositories…