3 citations · 8 across the 17 of their papers we have counts for
17 papers
Non-vacuous Generalization Bounds for Reinforcement Learning with Verifiable Rewards
Yuxuan Zhu, Rohan Alur, Daniel Kang
While reinforcement learning with verifiable rewards (RLVR) is widely used to improve the reasoning capabilities of large language models (LLMs), the generalizability of the result…
Behavioral Fingerprints for LLM Endpoint Stability and Identity
Jonah Leshin, Manish Shah, Ian Timmis +1
The consistency of AI-native applications depends on the behavioral consistency of the model endpoints that power them. Traditional reliability metrics such as uptime, latency and…
SODIUM: From Open Web Data to Queryable Databases
Chuxuan Hu, Philip Li, Maxwell Yang +1
During research, domain experts often ask analytical questions whose answers require integrating data from a wide range of web sources. Thus, they must spend substantial effort sea…
Accelerating Approximate Analytical Join Queries over Unstructured Data with Statistical Guarantees
Yuxuan Zhu, Tengjun Jin, Chenghao Mo +1
Analytical join queries over unstructured data are increasingly prevalent in data analytics. Applying machine learning (ML) models to label every pair in the cross product of table…
Human-Level Text-to-SQL via Reinforcement Learning on Verified Data, Without Pipeline Engineering
Yuxuan Zhu, Tengjun Jin, Yoojin Choi +1
Translating natural language questions to SQL queries (Text-to-SQL) is a long-standing problem in database research. Recent efforts have focused on improving accuracy by building i…
Noisy Data is Destructive to Reinforcement Learning with Verifiable Rewards
Yuxuan Zhu, Daniel Kang
Reinforcement learning with verifiable rewards (RLVR) has driven recent capability advances of large language models across various domains. Recent studies suggest that improved RL…
Pervasive Annotation Errors Break Text-to-SQL Benchmarks and Leaderboards
Tengjun Jin, Yoojin Choi, Yuxuan Zhu +1
Researchers have proposed numerous text-to-SQL techniques to streamline data analytics and accelerate the development of data-driven applications. To compare these techniques and s…
Measuring Agents in Production
Melissa Z. Pan, Negar Arabzadeh, Riccardo Cogo +22
LLM-based agents already operate in production across many industries, yet we lack an understanding of what technical methods make deployments successful. We present the first syst…
AIA Forecaster: Technical Report
Rohan Alur, Bradly C. Stadie, Daniel Kang +11
This technical report describes the AIA Forecaster, a Large Language Model (LLM)-based system for judgmental forecasting using unstructured data. The AIA Forecaster approach combin…
Analytical Queries for Unstructured Data
Daniel Kang
Unstructured data, in the form of text, images, video, and audio, is produced at exponentially higher rates. In tandem, machine learning (ML) methods have become increasingly power…
Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation
Sayash Kapoor, Benedikt Stroebl, Peter Kirgis +28
AI agents have been developed for complex real-world tasks from coding to customer service. But AI agent evaluations suffer from many challenges that undermine our understanding of…
Establishing Best Practices for Building Rigorous Agentic Benchmarks
Yuxuan Zhu, Tengjun Jin, Yada Pruksachatkun +22
Benchmarks are essential for quantitatively tracking progress in AI. As AI agents become increasingly capable, researchers and practitioners have introduced agentic benchmarks to e…
UTBoost: Rigorous Evaluation of Coding Agents on SWE-Bench
Boxi Yu, Yuxuan Zhu, Pinjia He +1
The advent of Large Language Models (LLMs) has spurred the development of coding agents for real-world code generation. As a widely used benchmark for evaluating the code generatio…
PandasBench: A Benchmark for the Pandas API
Alex Broihier, Stefanos Baziotis, Daniel Kang +1
The Pandas API has been central to the success of pandas and its alternatives. Despite its importance, there is no benchmark for it, and we argue that we cannot repurpose existing…
ELT-Bench: An End-to-End Benchmark for Evaluating AI Agents on ELT Pipelines
Tengjun Jin, Yuxuan Zhu, Daniel Kang
Practitioners are increasingly turning to Extract-Load-Transform (ELT) pipelines with the widespread adoption of cloud data warehouses. However, designing these pipelines often inv…
PilotDB: Database-Agnostic Online Approximate Query Processing with A Priori Error Guarantees (Technical Report)
Yuxuan Zhu, Tengjun Jin, Stefanos Baziotis +3
After decades of research in approximate query processing (AQP), its adoption in the industry remains limited. Existing methods struggle to simultaneously provide user-specified er…
CVE-Bench: A Benchmark for AI Agents' Ability to Exploit Real-World Web Application Vulnerabilities
Yuxuan Zhu, Antony Kellermann, Dylan Bowman +13
Large language model (LLM) agents are increasingly capable of autonomously conducting cyberattacks, posing significant threats to existing applications. This growing risk highlight…