5 papers
From High Recall to High Utility: Dataset-Adaptive Post-Processing of LLM-Generated Customer Intents
Mahesh Viswanathan, Joan Rossello, Leticia Fernandes +1
Large language models can extract useful signals from heterogeneous enterprise data, but high-recall extraction often produces outputs that are duplicated, uneven in granularity, s…
Evaluating Escalation Signals for LLM Routing: Targets, Controls, and Five Ways to Fool Yourself
Ramin Pishehvar, Andrea Morandi, Mahesh Viswanathan
Deciding when to escalate a query from a small language model to a larger one requires a cheap signal that predicts, before the large model is called, whether escalating would help…
Correcting Selection Bias in Sparse User Feedback for Large Language Model Quality Estimation: A Multi-Agent Hierarchical Bayesian Approach
Andrea Morandi, Mahesh Viswanathan
[Abridged] Production LLM deployments receive feedback from a non-random fraction of users: thumbs sit mostly in the tails of the satisfaction distribution, and a naive average ove…
From Intent to Execution: Composing Agentic Workflows with Agent Recommendation
Kishan Athrey, Ramin Pishehvar, Brian Riordan +1
Multi-Agent Systems (MAS) built using AI agents fulfill a variety of user intents that may be used to design and build a family of related applications. However, the creation of su…
Searching for Optimal Runtime Assurance via Reachability and Reinforcement Learning
Kristina Miller, Christopher K. Zeitler, William Shen +4
A runtime assurance system (RTA) for a given plant enables the exercise of an untrusted or experimental controller while assuring safety with a backup (or safety) controller. The r…