5 citations · 18 across the 24 of their papers we have counts for
27 papers
Stop Guessing When to Stop Testing: Efficient Model Evaluation with Just Enough Data
Ofir Arviv, Kristjan Greenewald, Yotam Perlitz +3
The inherent rigidity of fixed-size benchmarks makes them an inefficient tool for model evaluation. Diverse evaluation objectives, including model ranking, model selection and test…
AgentBeats: Agentifying Agent Assessment for Openness, Standardization, and Reproducibility
Xiaoyuan Liu, Jianhong Tu, Yuqi Chen +26
Agent systems are advancing quickly across domains, but their evaluation remains fragmented. Most benchmarks rely on fixed, LLM-centric harnesses that require heavy integration, cr…
A Matter of TASTE: Improving Coverage and Difficulty of Agent Benchmarks
Tomer Keren, Nitay Calderon, Asaf Yehudai +3
As agent capabilities advance, existing benchmarks, such as -Bench, are becoming increasingly saturated. Yet constructing new benchmark tasks remains complex, costly, and labo…
Agentic CLEAR: Automating Multi-Level Evaluation of LLM Agents
Asaf Yehudai, Lilach Eden, Michal Shmueli-Scheuer
Agentic systems are becoming more capable: agents define strategies, take actions, and interact with different environments. This autonomy poses serious challenges for overseeing a…
Time to REFLECT: Can We Trust LLM Judges for Evidence-based Research Agents?
Leyao Wang, Yanan He, Peng Chen +5
Deep research agents increasingly automate complex information-seeking tasks, producing evidence-grounded reports via multi-step reasoning, tool use, and synthesis. Their growing r…
CUBE: A Standard for Unifying Agent Benchmarks
Alexandre Lacoste, Nicolas Gontier, Oleh Shliazhko +23
The proliferation of agent benchmarks has created critical fragmentation that threatens research productivity. Each new benchmark requires substantial custom integration, creating…
ErrorMap and ErrorAtlas: Charting the Failure Landscape of Large Language Models
Shir Ashury-Tahan, Yifan Mai, Elron Bandel +2
Large Language Models (LLM) benchmarks tell us when models fail, but not why they fail. A wrong answer on a reasoning dataset may stem from formatting issues, calculation errors, o…
Robustness as an Emergent Property of Task Performance
Shir Ashury-Tahan, Ariel Gera, Elron Bandel +2
Robustness is widely viewed as a key challenge for real-world applications. However, because current research focuses only on difficult tasks, it partially captures real-world read…
General Agent Evaluation
Elron Bandel, Asaf Yehudai, Lilach Eden +12
General-purpose agents perform tasks in unfamiliar environments without domain-specific manual customization. Yet no study has systematically measured how agent architecture shapes…
CLEAR: Error Analysis via LLM-as-a-Judge Made Easy
Asaf Yehudai, Lilach Eden, Yotam Perlitz +2
The evaluation of Large Language Models (LLMs) increasingly relies on other LLMs acting as judges. However, current evaluation paradigms typically yield a single score or ranking,…
Think Again! The Effect of Test-Time Compute on Preferences, Opinions, and Beliefs of Large Language Models
George Kour, Itay Nakash, Ateret Anaby-Tavor +1
As Large Language Models (LLMs) become deeply integrated into human life and increasingly influence decision-making, it's crucial to evaluate whether and to what extent they exhibi…
Survey on Evaluation of LLM-based Agents
Asaf Yehudai, Lilach Eden, Alan Li +5
LLM-based agents represent a paradigm shift in AI, enabling autonomous systems to plan, reason, and use tools while interacting with dynamic environments. This paper provides the f…
DOVE: A Large-Scale Multi-Dimensional Predictions Dataset Towards Meaningful LLM Evaluation
Eliya Habba, Ofir Arviv, Itay Itzhak +5
Recent work found that LLMs are sensitive to a wide range of arbitrary prompt dimensions, including the type of delimiters, answer enumerators, instruction wording, and more. This…
The Mighty ToRR: A Benchmark for Table Reasoning and Robustness
Shir Ashury-Tahan, Yifan Mai, Rajmohan C +8
Despite its real-world significance, model performance on tabular data remains underexplored, leaving uncertainty about which model to rely on and which prompt configuration to ado…
Stay Tuned: An Empirical Study of the Impact of Hyperparameters on LLM Tuning in Real-World Applications
Alon Halfon, Shai Gretz, Ofir Arviv +6
Fine-tuning Large Language Models (LLMs) is an effective method to enhance their performance on downstream tasks. However, choosing the appropriate setting of tuning hyperparameter…
Do These LLM Benchmarks Agree? Fixing Benchmark Evaluation with BenchBench
Yotam Perlitz, Ariel Gera, Ofir Arviv +5
Recent advancements in Language Models (LMs) have catalyzed the creation of multiple benchmarks, designed to assess these models' general capabilities. A crucial task, however, is…
Unitxt: Flexible, Shareable and Reusable Data Preparation and Evaluation for Generative AI
Elron Bandel, Yotam Perlitz, Elad Venezian +9
In the dynamic landscape of generative NLP, traditional text processing pipelines limit research flexibility and reproducibility, as they are tailored to specific dataset, task, an…
Efficient Benchmarking of Language Models
Yotam Perlitz, Elron Bandel, Ariel Gera +6
The increasing versatility of language models (LMs) has given rise to a new class of benchmarks that comprehensively assess a broad range of capabilities. Such benchmarks are assoc…
Active Learning for Natural Language Generation
Yotam Perlitz, Ariel Gera, Michal Shmueli-Scheuer +3
The field of Natural Language Generation (NLG) suffers from a severe shortage of labeled data due to the extremely expensive and time-consuming process involved in manual annotatio…
nBIIG: A Neural BI Insights Generation System for Table Reporting
Yotam Perlitz, Dafna Sheinwald, Noam Slonim +1
We present nBIIG, a neural Business Intelligence (BI) Insights Generation system. Given a table, our system applies various analyses to create corresponding RDF representations, an…
Diversity Enhanced Table-to-Text Generation via Type Control
Yotam Perlitz, Liat Ein-Dor, Dafna Sheinwald +2
Generating natural language statements to convey logical inferences from tabular data (i.e., Logical NLG) is a process with one input and a variety of valid outputs. This character…
Quality Controlled Paraphrase Generation
Elron Bandel, Ranit Aharonov, Michal Shmueli-Scheuer +3
Paraphrase generation has been widely used in various downstream tasks. Most tasks benefit mainly from high quality paraphrases, namely those that are semantically similar to, yet…
HowSumm: A Multi-Document Summarization Dataset Derived from WikiHow Articles
Odellia Boni, Guy Feigenblat, Guy Lev +3
We present HowSumm, a novel large-scale dataset for the task of query-focused multi-document summarization (qMDS), which targets the use-case of generating actionable instructions…
orgFAQ: A New Dataset and Analysis on Organizational FAQs and User Questions
Guy Lev, Michal Shmueli-Scheuer, Achiya Jerbi +1
Frequently Asked Questions (FAQ) webpages are created by organizations for their users. FAQs are used in several scenarios, e.g., to answer user questions. On the other hand, the c…