8 citations · 21 across the 23 of their papers we have counts for
8 papers · 1 filter
Closing the Cross-Dialect Gap: Query Plans as a Portable Interface in Text-to-SQL
Corentin Royer, Robin Oester, Yotam Perlitz +3
Text-to-SQL systems are typically trained and evaluated on a single dialect (SQLite), yet production deployments span PostgreSQL, MySQL, ClickHouse, and beyond. We show that this s…
DataKernelBench: Can LLMs Optimize Database Queries on GPUs?
Gokul Karthik Kumar, Yotam Perlitz, Corey Lammie +2
GPUs increasingly accelerate database systems, but query-specific peak performance still often relies on hand-written kernels. Existing LLM kernel benchmarks focus on machine learn…
Stop Guessing When to Stop Testing: Efficient Model Evaluation with Just Enough Data
Ofir Arviv, Kristjan Greenewald, Yotam Perlitz +3
The inherent rigidity of fixed-size benchmarks makes them an inefficient tool for model evaluation. Diverse evaluation objectives, including model ranking, model selection and test…
A Matter of TASTE: Improving Coverage and Difficulty of Agent Benchmarks
Tomer Keren, Nitay Calderon, Asaf Yehudai +3
As agent capabilities advance, existing benchmarks, such as -Bench, are becoming increasingly saturated. Yet constructing new benchmark tasks remains complex, costly, and labo…
Instructions Shape Production of Language, not Processing
Andreas Waldis, Leshem Choshen, Yufang Hou +1
Instructions trigger a production-centered mechanism in language models. Through a cognitively inspired lens that separates language processing and production, we reveal this mecha…
PolySQL: Scaling Text-to-SQL Evaluation Across SQL Dialects via Automated Backend Isomorphism
Yotam Perlitz, Elad Venezian, Corentin Royer +2
SQL dialects vary in syntax, types, and functions across database engines. Text-to-SQL benchmarks, however, predominantly support only SQLite. This creates a critical evaluation ga…
ELT-Bench-Verified: Benchmark Quality Issues Underestimate AI Agent Capabilities
Christopher Zanoli, Andrea Giovannini, Tengjun Jin +2
Constructing Extract-Load-Transform (ELT) pipelines is a labor-intensive data engineering task and a high-impact target for AI automation. On ELT-Bench, the first benchmark for end…
General Agent Evaluation
Elron Bandel, Asaf Yehudai, Lilach Eden +12
General-purpose agents perform tasks in unfamiliar environments without domain-specific manual customization. Yet no study has systematically measured how agent architecture shapes…