Testing for LLM response differences: the case of a composite null consisting of semantically irrelevant query perturbations

Aranyak Acharyya, Carey E. Priebe, Hayden S. Helm

Given an input query, generative models such as large language models produce a random response drawn from a response distribution. Given two input queries, it is natural to ask if…