2 citations · 3 across the 9 of their papers we have counts for
6 papers · 1 filter
PLSQLBench: Benchmarking LLM Systems for Executable Procedural Database Programming
Marianne Menglin Liu, Leonid Boytsov, Daniel W. Peterson +13
We present PLSQLBench, to our knowledge the first benchmark for evaluating whether LLMs can write executable PL/SQL programs, with correctness measured through execution-based test…
GSM-SEM: Benchmark and Framework for Generating Semantically Variant Augmentations
Jyotika Singh, Fang Tu, Aziza Mirsaidova +11
Benchmarks like GSM8K are popular measures of mathematical reasoning, but leaderboard gains can overstate true capability due to memorization of fixed test sets. Most robustness va…
SPENCE: A Syntactic Probe for Detecting Contamination in NL2SQL Benchmarks
Mohammadtaher Safarzadeh, Hitesh Laxmichand Patel, Afshin Orojlooyjadid +2
Large language models (LLMs) have achieved strong performance on natural language to SQL (NL2SQL) benchmarks, yet their reported accuracy may be inflated by contamination from benc…
Play Favorites: A Statistical Method to Measure Self-Bias in LLM-as-a-Judge
Evangelia Spiliopoulou, Riccardo Fogliato, Hanna Burnsky +4
Large language models (LLMs) can serve as judges that offer rapid and reliable assessments of other LLM outputs. However, models may systematically assign overly favorable ratings…
MetaSynth: Meta-Prompting-Driven Agentic Scaffolds for Diverse Synthetic Data Generation
Haris Riaz, Sourav Bhabesh, Vinayak Arannil +2
Recent smaller language models such Phi-3.5 and Phi-4 rely on synthetic data generated using larger Language models. Questions remain about leveraging synthetic data for other use…
DoPAMine: Domain-specific Pre-training Adaptation from seed-guided data Mining
Vinayak Arannil, Neha Narwal, Sourav Sanjukta Bhabesh +5
Large Language Models (LLMs) have shown remarkable ability to generalize effectively across numerous industry domains while executing a range of tasks. Many of these competencies a…