6 papers
Challenges in Testing Large Language Model Based Software: A Faceted Taxonomy
Felix Dobslaw, Robert Feldt, Juyeon Yoon +1
Large Language Models (LLMs) and Multi-Agent LLMs (MALLMs) introduce non-determinism unlike traditional or machine learning software, requiring new approaches to verifying correctn…
Predictive Prompt Analysis
Jae Yong Lee, Sungmin Kang, Shin Yoo
Large Language Models (LLMs) are machine learning models that have seen widespread adoption due to their capability of handling previously difficult tasks. LLMs, due to their train…
COSMosFL: Ensemble of Small Language Models for Fault Localisation
Hyunjoon Cho, Sungmin Kang, Gabin An +1
LLMs are rapidly being adopted to build powerful tools and agents for software engineering, but most of them rely heavily on extremely large closed-source models. This, in turn, ca…
Adaptive Testing for LLM-Based Applications: A Diversity-based Approach
Juyeon Yoon, Robert Feldt, Shin Yoo
The recent surge of building software systems powered by Large Language Models (LLMs) has led to the development of various testing frameworks, primarily focused on treating prompt…
Lachesis: Predicting LLM Inference Accuracy using Structural Properties of Reasoning Paths
Naryeong Kim, Sungmin Kang, Gabin An +1
Large Language Models are increasingly used to build agents to perform more complex tasks. As LLMs perform more complicated reasoning through longer interactions, self-consistency,…
Beyond pip install: Evaluating LLM Agents for the Automated Installation of Python Projects
Louis Milliken, Sungmin Kang, Shin Yoo
Many works have recently proposed the use of Large Language Model (LLM) based agents for performing `repository level' tasks, loosely defined as a set of tasks whose scopes are gre…