1 citations · 3 across the 11 of their papers we have counts for
11 papers
Can Agents Infer Environment from Interaction? Evidence from Agentic Automata Learning
Reef Menaged, Gili Lior, Shauli Ravfogel +2
We propose agentic automata learning to evaluate the extent to which tool-calling LLM agents can uncover hidden environments through interaction, a capability increasingly required…
Extending Item Response Theory for Efficient and Meaningful Multilingual Evaluation
Gili Lior, Tzviel Frostig, Gabriel Stanovsky +1
Multilingual benchmarks are central to evaluating large language models (LLMs) across languages, but they suffer from three issues: exhaustive evaluation scales linearly with the n…
From Feelings to Metrics: Understanding and Formalizing How Users Vibe-Test LLMs
Itay Itzhak, Eliya Habba, Gabriel Stanovsky +1
Evaluating LLMs is challenging, as benchmark scores often fail to capture models' real-world usefulness. Instead, users often rely on ``vibe-testing'': informal experience-based ev…
ScheMatiQ: From Research Question to Structured Data through Interactive Schema Discovery
Shahar Levy, Eliya Habba, Reshef Mintz +3
Many disciplines pose natural-language research questions over large document collections whose answers typically require structured evidence, traditionally obtained by manually de…
Anticipatory Evaluation of Language Models: Predicting LLM Benchmark Scores from Descriptions
Jungsoo Park, Ethan Mendes, Gabriel Stanovsky +1
Modern ML research increasingly faces an evaluation bottleneck: building benchmarks and running experiments is costly, slowing iteration. We ask how much can be inferred about eval…
PromptSuite: A Task-Agnostic Framework for Multi-Prompt Generation
Eliya Habba, Noam Dahan, Gili Lior +1
Evaluating LLMs with a single prompt has proven unreliable, with small changes leading to significant performance differences. However, generating the prompt variations needed for…
ReliableEval: A Recipe for Stochastic LLM Evaluation via Method of Moments
Gili Lior, Eliya Habba, Shahar Levy +2
LLMs are highly sensitive to prompt phrasing, yet standard benchmarks typically report performance using a single prompt, raising concerns about the reliability of such evaluations…
DOVE: A Large-Scale Multi-Dimensional Predictions Dataset Towards Meaningful LLM Evaluation
Eliya Habba, Ofir Arviv, Itay Itzhak +5
Recent work found that LLMs are sensitive to a wide range of arbitrary prompt dimensions, including the type of delimiters, answer enumerators, instruction wording, and more. This…
Beyond Benchmarks: On The False Promise of AI Regulation
Gabriel Stanovsky, Renana Keydar, Gadi Perl +1
The performance of AI models on safety benchmarks does not indicate their real-world performance after deployment. This opaqueness of AI models impedes existing regulatory framewor…
The Perfect Victim: Computational Analysis of Judicial Attitudes towards Victims of Sexual Violence
Eliya Habba, Renana Keydar, Dan Bareket +1
We develop computational models to analyze court statements in order to assess judicial attitudes toward victims of sexual violence in the Israeli court system. The study examines…
Automated Extraction of Sentencing Decisions from Court Cases in the Hebrew Language
Mohr Wenger, Tom Kalir, Noga Berger +3
We present the task of Automated Punishment Extraction (APE) in sentencing decisions from criminal court cases in Hebrew. Addressing APE will enable the identification of sentencin…