4 papers
MCP-Atlas: A Large-Scale Benchmark for Tool-Use Competency with Real MCP Servers
Chaithanya Bandi, Razvan-Gabriel Dumitru, Ben Hertzberg +20
The Model Context Protocol (MCP) is emerging as a standard interface through which large language model (LLM) agents discover and invoke external tools. However, existing MCP evalu…
OpenJarvis: Personal AI, On Personal Devices
Jon Saad-Falcon, Avanika Narayan, Robby Manihani +10
Personal AI stacks, like OpenClaw and Hermes Agent, are becoming central to daily work, yet they route nearly every query (often over sensitive local data) to cloud-hosted frontier…
SWE Atlas: Benchmarking Coding Agents Beyond Issue Resolution
Mohit Raghavendra, Soham Dan, Miguel Romero Calvo +12
We introduce SWE Atlas, a benchmark suite for coding agents spanning three professional software engineering workflows: Codebase Q&A (124 tasks), Test Writing (90 tasks), and Refac…
SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?
Xiang Deng, Jeff Da, Edwin Pan +19
We introduce SWE-Bench Pro, a substantially more challenging benchmark that builds upon the best practices of SWE-BENCH [25], but is explicitly designed to capture realistic, compl…