12 papers
MetaExp: Learning to Self-Improve through Experience via Verifiable Cross-Task Environment Scaling
Renhao Li, Xinjie Shen, Yang Su +4
Language-model agents can accumulate interaction feedback, but reliably turning that feedback into reusable experience remains challenging. Existing methods either optimize externa…
CyberArena: Benchmarking AI Agents in Live Symmetric Attack-and-Defense Combat
Bangya Liu, Jianhong Tu, Yueyang Pan +18
AI agents are demonstrating increasingly strong offensive and defensive cybersecurity capabilities in the real world. Current benchmarks mostly evaluate agents in isolation against…
D2K-Bench: Can LLM Agents Turn Expert Designs into Efficient GPU Kernels?
Daifeng Li, Huiqiang Jiang, Chengruidong Zhang +6
GPU kernels generated by large language model (LLM) agents can remain less efficient than expert implementations, but runtime alone does not reveal how the gap relates to design di…
Verifiable Hidden Dynamics Play: Generating Agentic RL Environments from Solved Mechanisms
Xinjie Shen, Wei Fan, Xudong Guo +5
Language-model agents increasingly face long-horizon tasks with evolving state, interdependent decisions, and delayed outcomes. Scaling their training requires diverse agentic envi…
BabelArena: A Large-Scale Multilingual Benchmark for LLM Agents
Peng Kuang, Yuchun Fan, Jiangnan Li +7
Large language model (LLM) agents increasingly execute multi-step workflows through tool use and interaction with users and environments. However, current agent evaluations are lar…
E-Commerce Bench: Evaluating LLM Agents on Long-Horizon Autonomous Business Operation
Wei Fan, Xinjie Shen, Xudong Guo +8
Long-horizon agentic tasks go beyond chaining short tasks over more interaction turns. Their evolving dynamic environments and long-range dependencies require Large Language Models…
CyberGym-E2E: Scalable Real-World Benchmark for AI Agents' End-to-End Cybersecurity Capabilities
Tianneng Shi, Robin Rheem, Dongwei Jiang +13
AI has the potential to transform cybersecurity by enabling systems that can autonomously detect, analyze, and remediate software vulnerabilities. However, existing cybersecurity e…
Qwen-AgentWorld: Language World Models for General Agents
Yuxin Zuo, Zikai Xiao, Li Sheng +30
A world model predicts environment dynamics based on current observations and actions, serving as a core cognitive mechanism for reasoning and planning. In this work, we investigat…
AgentBeats: Agentifying Agent Assessment for Openness, Standardization, and Reproducibility
Xiaoyuan Liu, Jianhong Tu, Yuqi Chen +26
Agent systems are advancing quickly across domains, but their evaluation remains fragmented. Most benchmarks rely on fixed, LLM-centric harnesses that require heavy integration, cr…
Agents' Last Exam
Yiyou Sun, Xinyang Han, Weichen Zhang +378
Recent AI systems have achieved strong results on a wide range of benchmarks, yet these gains have not translated into economically meaningful deployment across many professional d…
Predicting Task Performance with Context-aware Scaling Laws
Kyle Montgomery, David Park, Jianhong Tu +4
Scaling laws have transformed our understanding of large language models by linking upstream metrics like cross-entropy loss to design factors such as model size, training data, an…
MLAN: Language-Based Instruction Tuning Preserves and Transfers Knowledge in Multimodal Language Models
Jianhong Tu, Zhuohao Ni, Nicholas Crispino +8
We present a novel visual instruction tuning strategy to improve the zero-shot task generalization of multimodal large language models by building a firm text-only knowledge base.…