26 citations · 50 across the 24 of their papers we have counts for
25 papers
PACIFIC: a framework for generating benchmarks to check Precise Automatically Checked Instruction Following In Code
Itay Dreyfuss, Antonio Abu Nassar, Samuel Ackerman +5
Large Language Model (LLM)-based code assistants have emerged as a powerful application of generative AI, demonstrating impressive capabilities in code generation and comprehension…
Uncovering Code Insights: Leveraging GitHub Artifacts for Deeper Code Understanding
Ziv Nevo, Orna Raz, Karen Yorav
Understanding the purpose of source code is a critical task in software maintenance, onboarding, and modernization. While large language models (LLMs) have shown promise in generat…
Evaluating perturbation robustness of generative systems that use COBOL code inputs
Samuel Ackerman, Wesam Ibraheem, Orna Raz +1
Systems incorporating large language models (LLMs) as a component are known to be sensitive (i.e., non-robust) to minor input variations that do not change the meaning of the input…
Vintage Code, Modern Judges: Meta-Validation in Low Data Regimes
Ora Nova Fandina, Gal Amram, Eitan Farchi +6
Application modernization in legacy languages such as COBOL, PL/I, and REXX faces an acute shortage of resources, both in expert availability and in high-quality human evaluation d…
Automated Validation of LLM-based Evaluators for Software Engineering Artifacts
Ora Nova Fandina, Eitan Farchi, Shmulik Froimovich +4
Automation in software engineering increasingly relies on large language models (LLMs) to generate, review, and assess code artifacts. However, establishing LLMs as reliable evalua…
Statistical multi-metric evaluation and visualization of LLM system predictive performance
Samuel Ackerman, Eitan Farchi, Orna Raz +1
The evaluation of generative or discriminative large language model (LLM)-based systems is often a complex multi-dimensional problem. Typically, a set of system configuration alter…
Automatic Generation of Benchmarks and Reliable LLM Judgment for Code Tasks
Eitan Farchi, Shmulik Froimovich, Rami Katan +1
LLMs can be used in a variety of code related tasks such as translating from one programming language to another, implementing natural language requirements and code summarization.…
How Safe is Your Safety Metric? Automatic Concatenation Tests for Metric Reliability
Ora Nova Fandina, Leshem Choshen, Eitan Farchi +3
Consider a scenario where a harmfulness evaluation metric intended to filter unsafe responses from a Large Language Model. When applied to individual harmful prompt-response pairs,…
Generating Unseen Code Tests In Infinitum
Marcel Zalmanovici, Orna Raz, Eitan Farchi +1
Large Language Models (LLMs) are used for many tasks, including those related to coding. An important aspect of being able to utilize LLMs is the ability to assess their fitness fo…
Using Combinatorial Optimization to Design a High quality LLM Solution
Samuel Ackerman, Eitan Farchi, Rami Katan +1
We introduce a novel LLM based solution design approach that utilizes combinatorial optimization and sampling. Specifically, a set of factors that influence the quality of the solu…
Alignment Studio: Aligning Large Language Models to Particular Contextual Regulations
Swapnaja Achintalwar, Ioana Baldini, Djallel Bouneffouf +16
The alignment of large language models is usually done by model providers to add or control behaviors that are common or universally understood across use cases and contexts. In co…
Detectors for Safe and Reliable LLMs: Implementations, Uses, and Limitations
Swapnaja Achintalwar, Adriana Alvarado Garcia, Ateret Anaby-Tavor +35
Large language models (LLMs) are susceptible to a variety of risks, from non-faithful output to biased and toxic generations. Due to several limiting factors surrounding LLMs (trai…
Unveiling Safety Vulnerabilities of Large Language Models
George Kour, Marcel Zalmanovici, Naama Zwerdling +5
As large language models become more prevalent, their possible harmful or inappropriate responses are a cause for concern. This paper introduces a unique dataset containing adversa…
Predicting Question-Answering Performance of Large Language Models through Semantic Consistency
Ella Rabinovich, Samuel Ackerman, Orna Raz +2
Semantic consistency of a language model is broadly defined as the model's ability to produce semantically-equivalent outputs, given semantically-equivalent inputs. We address the…
Automatic Generation of Attention Rules For Containment of Machine Learning Model Errors
Samuel Ackerman, Axel Bendavid, Eitan Farchi +1
Machine learning (ML) solutions are prevalent in many applications. However, many challenges exist in making these solutions business-grade. For instance, maintaining the error rat…
Measuring the Measuring Tools: An Automatic Evaluation of Semantic Metrics for Text Corpora
George Kour, Samuel Ackerman, Orna Raz +3
The ability to compare the semantic similarity between text corpora is important in a variety of natural language processing applications. However, standard methods for evaluating…
High-quality Conversational Systems
Samuel Ackerman, Ateret Anaby-Tavor, Eitan Farchi +7
Conversational systems or chatbots are an example of AI-Infused Applications (AIIA). Chatbots are especially important as they are often the first interaction of clients with a bus…
Theory and Practice of Quality Assurance for Machine Learning Systems An Experiment Driven Approach
Samuel Ackerman, Guy Barash, Eitan Farchi +2
The crafting of machine learning (ML) based systems requires statistical control throughout its life cycle. Careful quantification of business requirements and identification of ke…
Classifier Data Quality: A Geometric Complexity Based Method for Automated Baseline And Insights Generation
George Kour, Marcel Zalmanovici, Orna Raz +2
Testing Machine Learning (ML) models and AI-Infused Applications (AIIAs), or systems that contain ML models, is highly challenging. In addition to the challenges of testing classic…
Automatically detecting data drift in machine learning classifiers
Samuel Ackerman, Orna Raz, Marcel Zalmanovici +1
Classifiers and other statistics-based machine learning (ML) techniques generalize, or learn, based on various statistical properties of the training data. The assumption underlyin…
Density-based interpretable hypercube region partitioning for mixed numeric and categorical data
Samuel Ackerman, Eitan Farchi, Orna Raz +2
Consider a structured dataset of features, such as . A user may want to know where in the feature space obser…
Detecting model drift using polynomial relations
Eliran Roffe, Samuel Ackerman, Orna Raz +1
Machine learning models serve critical functions, such as classifying loan applicants as good or bad risks. Each model is trained under the assumption that the data used in trainin…
FreaAI: Automated extraction of data slices to test machine learning models
Samuel Ackerman, Orna Raz, Marcel Zalmanovici
Machine learning (ML) solutions are prevalent. However, many challenges exist in making these solutions business-grade. One major challenge is to ensure that the ML solution provid…
Machine Learning Model Drift Detection Via Weak Data Slices
Samuel Ackerman, Parijat Dube, Eitan Farchi +2
Detecting drift in performance of Machine Learning (ML) models is an acknowledged challenge. For ML models to become an integral part of business applications it is essential to de…