6 papers
MENTOR: Procedural Shielding of Tool-Using LLM Agents
Ashwani Anand, Ivi Chatzi, Ritam Raha +1
Tool-using LLM agents are increasingly used to automate workflows governed by procedural manuals, such as company rules for booking travel or procuring hardware. These manuals are…
MANTRA: Synthesizing SMT-Validated Compliance Benchmarks for Tool-Using LLM Agents
Ashwani Anand, Ivi Chatzi, Ritam Raha +1
Tool-using large language model (LLM) agents are increasingly deployed in settings where their reliable behavior is governed by strict procedural manuals. Ensuring that such agents…
Tokenization Multiplicity Leads to Arbitrary Price Variation in LLM-as-a-service
Ivi Chatzi, Nina Corvelo Benz, Stratis Tsirtsis +1
Providers of LLM-as-a-service have predominantly adopted a simple pricing model: users pay a fixed price per token. Consequently, one may think that the price two different users w…
Evaluation of Large Language Models via Coupled Token Generation
Nina Corvelo Benz, Stratis Tsirtsis, Eleni Straitouri +4
State of the art large language models rely on randomization to respond to a prompt. As an immediate consequence, a model may respond differently to the same prompt if asked multip…
Counterfactual Token Generation in Large Language Models
Ivi Chatzi, Nina Corvelo Benz, Eleni Straitouri +2
"Sure, I am happy to generate a story for you: Captain Lyra stood at the helm of her trusty ship, the Maelstrom's Fury, gazing out at the endless sea. [...] Lyra's eyes welled up w…
Prediction-Powered Ranking of Large Language Models
Ivi Chatzi, Eleni Straitouri, Suhas Thejaswi +1
Large language models are often ranked according to their level of alignment with human preferences -- a model is better than other models if its outputs are more frequently prefer…