26 citations · 53 across the 22 of their papers we have counts for
23 papers
PACIFIC: a framework for generating benchmarks to check Precise Automatically Checked Instruction Following In Code
Itay Dreyfuss, Antonio Abu Nassar, Samuel Ackerman +5
Large Language Model (LLM)-based code assistants have emerged as a powerful application of generative AI, demonstrating impressive capabilities in code generation and comprehension…
Evaluating perturbation robustness of generative systems that use COBOL code inputs
Samuel Ackerman, Wesam Ibraheem, Orna Raz +1
Systems incorporating large language models (LLMs) as a component are known to be sensitive (i.e., non-robust) to minor input variations that do not change the meaning of the input…
Statistical multi-metric evaluation and visualization of LLM system predictive performance
Samuel Ackerman, Eitan Farchi, Orna Raz +1
The evaluation of generative or discriminative large language model (LLM)-based systems is often a complex multi-dimensional problem. Typically, a set of system configuration alter…
A Novel Metric for Measuring the Robustness of Large Language Models in Non-adversarial Scenarios
Samuel Ackerman, Ella Rabinovich, Eitan Farchi +1
We evaluate the robustness of several large language models on multiple datasets. Robustness here refers to the relative insensitivity of the model's answers to meaning-preserving…
Using Combinatorial Optimization to Design a High quality LLM Solution
Samuel Ackerman, Eitan Farchi, Rami Katan +1
We introduce a novel LLM based solution design approach that utilizes combinatorial optimization and sampling. Specifically, a set of factors that influence the quality of the solu…
Predicting Question-Answering Performance of Large Language Models through Semantic Consistency
Ella Rabinovich, Samuel Ackerman, Orna Raz +2
Semantic consistency of a language model is broadly defined as the model's ability to produce semantically-equivalent outputs, given semantically-equivalent inputs. We address the…