21 citations · 43 across the 30 of their papers we have counts for
30 papers
A Persona-based Rate Action Index
Hayden Helm, Andrew Dassori
We propose an index for predicting the U.S.\ Federal Open Market Committee (FOMC) decision to hike/hold/cut the current federal funds target rate based on how a collection of perso…
Query-efficient model evaluation using cached responses
Hayden Helm, Ben Johnson, Carey Priebe
Evaluating a new model on an existing benchmark is often necessary to understand its behavior before deployment. For modern evaluation frameworks, generating and evaluating a respo…
When prompt perturbations break your A/B test: A valid statistical test for generative surveying
Hayden Helm, Carey Priebe
Generative surveying -- where collections of LLM-based personas provide feedback on messages -- has emerged as a cheap and scalable alternative to traditional market research. Howe…
Jailbreak susceptibility prediction and mitigation via the behavioral geometry of models
Hayden Helm, Xiaodong Liu, Weiwei Yang
Evaluating and mitigating a generative system's susceptibility to jailbreak attacks is critical to its safe deployment. Given the number of deployable systems, full per-configurati…
Control Charts for Multi-agent Systems
Hayden Helm, Carey Priebe, Brandon Duderstadt
Generative agents have proven to be powerful assistants in a wide variety of contexts. Given this success, users are now deploying agents with minimal restrictions in open ended, m…
Black-box model classification under the discriminative factorization
Hayden Helm, Merrick Ohata, Carey Priebe
Access to modern generative systems is often restricted to querying an API (the ``black-box" setting) and many properties of the system are unknown to the user at inference time. W…
Detecting Perspective Shifts in Multi-agent Systems
Eric Bridgeford, Hayden Helm
Generative models augmented with external tools and update mechanisms (or \textit{agents}) have demonstrated capabilities beyond intelligent prompting of base models. As agent use…
Testing for LLM response differences: the case of a composite null consisting of semantically irrelevant query perturbations
Aranyak Acharyya, Carey E. Priebe, Hayden S. Helm
Given an input query, generative models such as large language models produce a random response drawn from a response distribution. Given two input queries, it is natural to ask if…
Toward a digital twin of U.S. Congress
Hayden Helm, Tianyi Chen, Harvey McGuinness +3
In this paper we provide evidence that a virtual model of U.S. congresspersons based on a collection of language models satisfies the definition of a digital twin. In particular, w…
Investigating social alignment via mirroring in a system of interacting language models
Harvey McGuinness, Tianyu Wang, Carey E. Priebe +1
Alignment is a social phenomenon wherein individuals share a common goal or perspective. Mirroring, or mimicking the behaviors and opinions of another individual, is one mechanism…
Statistical inference on black-box generative models in the data kernel perspective space
Hayden Helm, Aranyak Acharyya, Brandon Duderstadt +2
Generative models are capable of producing human-expert level content across a variety of topics and domains. As the impact of generative models grows, it is necessary to develop s…
Consistent estimation of generative model representations in the data kernel perspective space
Aranyak Acharyya, Michael W. Trosset, Carey E. Priebe +1
Generative models, such as large language models and text-to-image diffusion models, produce relevant information when presented a query. Different models may produce different inf…
Tracking the perspectives of interacting language models
Hayden Helm, Brandon Duderstadt, Youngser Park +1
Large language models (LLMs) are capable of producing high quality information at unprecedented rates. As these models continue to entrench themselves in society, the content they…
MedFuzz: Exploring the Robustness of Large Language Models in Medical Question Answering
Robert Osazuwa Ness, Katie Matton, Hayden Helm +4
Large language models (LLM) have achieved impressive performance on medical question-answering benchmarks. However, high benchmark accuracy does not imply that the performance gene…
A Statistical Turing Test for Generative Models
Hayden Helm, Carey E. Priebe, Weiwei Yang
The emergence of human-like abilities of AI systems for content generation in domains such as text, audio, and vision has prompted the development of classifiers to determine wheth…
Comparing Foundation Models using Data Kernels
Brandon Duderstadt, Hayden S. Helm, Carey E. Priebe
Recent advances in self-supervised learning and neural network scaling have enabled the creation of large models, known as foundation models, which can be easily adapted to a wide…
Approximately optimal domain adaptation with Fisher's Linear Discriminant
Hayden S. Helm, Ashwin De Silva, Joshua T. Vogelstein +2
We propose a class of models based on Fisher's Linear Discriminant (FLD) in the context of domain adaptation. The class is the convex combination of two hypotheses: i) an average h…
Deep Learning with Label Noise: A Hierarchical Approach
Li Chen, Ningyuan Huang, Cong Mu +4
Deep neural networks are susceptible to label noise. Existing methods to improve robustness, such as meta-learning and regularization, usually require significant change to the net…
Mental State Classification Using Multi-graph Features
Guodong Chen, Hayden S. Helm, Kate Lytvynets +2
We consider the problem of extracting features from passive, multi-channel electroencephalogram (EEG) devices for downstream inference tasks related to high-level mental states suc…
Towards a theory of out-of-distribution learning
Jayanta Dey, Ali Geisa, Ronak Mehta +7
Learning is a process wherein a learning agent enhances its performance through exposure of experience or data. Throughout this journey, the agent may encounter diverse learning en…
Leveraging semantically similar queries for ranking via combining representations
Hayden S. Helm, Marah Abdin, Benjamin D. Pedigo +8
In modern ranking problems, different and disparate representations of the items to be ranked are often available. It is sensible, then, to try to combine these representations to…
Inducing a hierarchy for multi-class classification problems
Hayden S. Helm, Weiwei Yang, Sujeeth Bharadwaj +5
In applications where categorical labels follow a natural hierarchy, classification methods that exploit the label structure often outperform those that do not. Un-fortunately, the…
Subgraph nomination: Query by Example Subgraph Retrieval in Networks
Al-Fahad M. Al-Qadhi, Carey E. Priebe, Hayden S. Helm +1
This paper introduces the subgraph nomination inference task, in which example subgraphs of interest are used to query a network for similarly interesting subgraphs. This type of p…
A partition-based similarity for classification distributions
Hayden S. Helm, Ronak D. Mehta, Brandon Duderstadt +5
Herein we define a measure of similarity between classification distributions that is both principled from the perspective of statistical pattern recognition and useful from the pe…