8 citations · 21 across the 23 of their papers we have counts for
17 papers · 1 filter
Closing the Cross-Dialect Gap: Query Plans as a Portable Interface in Text-to-SQL
Corentin Royer, Robin Oester, Yotam Perlitz +3
Text-to-SQL systems are typically trained and evaluated on a single dialect (SQLite), yet production deployments span PostgreSQL, MySQL, ClickHouse, and beyond. We show that this s…
DataKernelBench: Can LLMs Optimize Database Queries on GPUs?
Gokul Karthik Kumar, Yotam Perlitz, Corey Lammie +2
GPUs increasingly accelerate database systems, but query-specific peak performance still often relies on hand-written kernels. Existing LLM kernel benchmarks focus on machine learn…
Instructions Shape Production of Language, not Processing
Andreas Waldis, Leshem Choshen, Yufang Hou +1
Instructions trigger a production-centered mechanism in language models. Through a cognitively inspired lens that separates language processing and production, we reveal this mecha…
PolySQL: Scaling Text-to-SQL Evaluation Across SQL Dialects via Automated Backend Isomorphism
Yotam Perlitz, Elad Venezian, Corentin Royer +2
SQL dialects vary in syntax, types, and functions across database engines. Text-to-SQL benchmarks, however, predominantly support only SQLite. This creates a critical evaluation ga…
CLEAR: Error Analysis via LLM-as-a-Judge Made Easy
Asaf Yehudai, Lilach Eden, Yotam Perlitz +2
The evaluation of Large Language Models (LLMs) increasingly relies on other LLMs acting as judges. However, current evaluation paradigms typically yield a single score or ranking,…
NeurIPS 2023 LLM Efficiency Fine-tuning Competition
Mark Saroufim, Yotam Perlitz, Leshem Choshen +11
Our analysis of the NeurIPS 2023 large language model (LLM) fine-tuning competition revealed the following trend: top-performing models exhibit significant overfitting on benchmark…
DOVE: A Large-Scale Multi-Dimensional Predictions Dataset Towards Meaningful LLM Evaluation
Eliya Habba, Ofir Arviv, Itay Itzhak +5
Recent work found that LLMs are sensitive to a wide range of arbitrary prompt dimensions, including the type of delimiters, answer enumerators, instruction wording, and more. This…
The Mighty ToRR: A Benchmark for Table Reasoning and Robustness
Shir Ashury-Tahan, Yifan Mai, Rajmohan C +8
Despite its real-world significance, model performance on tabular data remains underexplored, leaving uncertainty about which model to rely on and which prompt configuration to ado…
JuStRank: Benchmarking LLM Judges for System Ranking
Ariel Gera, Odellia Boni, Yotam Perlitz +3
Given the rapid progress of generative AI, there is a pressing need to systematically compare and choose between the numerous models and configurations available. The scale and ver…
Do These LLM Benchmarks Agree? Fixing Benchmark Evaluation with BenchBench
Yotam Perlitz, Ariel Gera, Ofir Arviv +5
Recent advancements in Language Models (LMs) have catalyzed the creation of multiple benchmarks, designed to assess these models' general capabilities. A crucial task, however, is…
Holmes: A Benchmark to Assess the Linguistic Competence of Language Models
Andreas Waldis, Yotam Perlitz, Leshem Choshen +2
We introduce Holmes, a new benchmark designed to assess language models (LMs) linguistic competence - their unconscious understanding of linguistic phenomena. Specifically, we use…
Unitxt: Flexible, Shareable and Reusable Data Preparation and Evaluation for Generative AI
Elron Bandel, Yotam Perlitz, Elad Venezian +9
In the dynamic landscape of generative NLP, traditional text processing pipelines limit research flexibility and reproducibility, as they are tailored to specific dataset, task, an…
Efficient Benchmarking of Language Models
Yotam Perlitz, Elron Bandel, Ariel Gera +6
The increasing versatility of language models (LMs) has given rise to a new class of benchmarks that comprehensively assess a broad range of capabilities. Such benchmarks are assoc…
Active Learning for Natural Language Generation
Yotam Perlitz, Ariel Gera, Michal Shmueli-Scheuer +3
The field of Natural Language Generation (NLG) suffers from a severe shortage of labeled data due to the extremely expensive and time-consuming process involved in manual annotatio…
nBIIG: A Neural BI Insights Generation System for Table Reporting
Yotam Perlitz, Dafna Sheinwald, Noam Slonim +1
We present nBIIG, a neural Business Intelligence (BI) Insights Generation system. Given a table, our system applies various analyses to create corresponding RDF representations, an…
Zero-Shot Text Classification with Self-Training
Ariel Gera, Alon Halfon, Eyal Shnarch +3
Recent advances in large pretrained language models have increased attention to zero-shot text classification. In particular, models finetuned on natural language inference dataset…
Diversity Enhanced Table-to-Text Generation via Type Control
Yotam Perlitz, Liat Ein-Dor, Dafna Sheinwald +2
Generating natural language statements to convey logical inferences from tabular data (i.e., Logical NLG) is a process with one input and a variety of valid outputs. This character…