11 papers
CALICO: A Human-Centered, Codebook-Aligned System for Annotation
Boqin Yuan, Xiaoyi Gu, Fiona Li +3
Large language models are increasingly used to scale codebook-based annotation in scientific research, but existing workflows provide limited support for translating domain experts…
Harbor Adapters and Harbor-Index: Infrastructure and a Curated Meta-Dataset for Large-Scale Agentic Evaluation
Lin Shi, Haowei Lin, Zixuan Zhu +123
Evaluating agents on the growing number of agentic benchmarks is challenging because they often require complex environments and agent integrations. We introduce Harbor Adapters, a…
CodeNib: A Multi-View Data System for Serving Repository Context to Coding Agents
Zhongming Yu, Hengjia Yu, Boqin Yuan +12
An incremental code index can complete every update without error and still return the wrong graph half the time. Coding agents depend on such reused views for search and navigatio…
Agents' Last Exam
Yiyou Sun, Xinyang Han, Weichen Zhang +378
Recent AI systems have achieved strong results on a wide range of benchmarks, yet these gains have not translated into economically meaningful deployment across many professional d…
ClawTrace: Cost-Aware Tracing for LLM Agent Skill Distillation
Boqin Yuan, Yue Su, Renchu Song +2
Skill-distillation pipelines learn reusable rules from LLM agent trajectories, but they lack a key signal: how much each step costs. Without per-step cost, a pipeline cannot distin…
Diagnosing Retrieval vs. Utilization Bottlenecks in LLM Agent Memory
Boqin Yuan, Yue Su, Kun Yao
Memory-augmented LLM agents store and retrieve information from prior interactions, yet the relative importance of how memories are written versus how they are retrieved remains un…
AMA-Bench: Evaluating Long-Horizon Memory for Agentic Applications
Yujie Zhao, Boqin Yuan, Junbo Huang +9
Large Language Models (LLMs) are increasingly used as autonomous agents in complex, long-horizon applications, where effective memory is critical for sustained performance. Yet exi…
SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks
Xiangyi Li, Yimin Liu, Wenbo Chen +75
Agent Skills are structured packages of procedural knowledge that augment large language model (LLM) agents at inference time. Despite rapid adoption, there is no standard way to m…
L-MARS: Legal Multi-Agent System with Agentic Search and Citation-Faithfulness Audit
Boqin Yuan, Ziqi Wang
Large language models are increasingly deployed for legal question answering, where evaluations typically focus on multiple-choice accuracy. This measure overlooks a common failure…
aiXiv: A Next-Generation Open Access Ecosystem for Scientific Discovery Generated by AI Scientists
Pengsong Zhang, Xiang Hu, Guowei Huang +20
Recent advances in large language models (LLMs) have enabled AI agents to autonomously generate scientific proposals, conduct experiments, author papers, and perform peer reviews.…
PRO-V-R1: Reasoning Enhanced Programming Agent for RTL Verification
Yujie Zhao, Zhijing Wu, Boqin Yuan +6
Register-Transfer Level (RTL) verification is a primary bottleneck, consuming 60-70% of development time. While Large Language Models (LLMs) show promise for RTL automation, their…