8 papers
A Jagged Frontier: Evaluating Robustness of Code Agents to Semantics-Preserving Transformations
Hasan Najib Mahmud, Shreya Gupta, Isha Chaudhary +4
AI code agents are increasingly deployed to resolve real software issues, yet their reliability under superficial code variations remains poorly understood. We evaluate whether cod…
Quantitative Certification of Agentic Tool Selection
Jehyeok Yeon, Isha Chaudhary, Gagandeep Singh
Large language models (LLMs) are increasingly deployed in agentic systems, where a fundamental task is mapping user intents to relevant external tools. Errors in tool selection can…
Revealing Interpretable Failure Modes of VLMs
Isha Chaudhary, Vedaant V Jain, Kavya Sachdeva +2
Vision-Language Models (VLMs) are increasingly used in safety-critical applications because of their broad reasoning capabilities and ability to generalize with minimal task-specif…
Lumos: Let there be Language Model System Certification
Isha Chaudhary, Vedaant Jain, Prineet Parhar +4
We introduce the first principled framework, Lumos, for specifying and formally certifying Language Model System (LMS) behaviors. Lumos is an imperative probabilistic programming D…
How Catastrophic is Your LLM? Certifying Risk in Conversation
Chengxiao Wang, Isha Chaudhary, Qian Hu +3
Large Language Models (LLMs) can produce catastrophic responses in conversational settings that pose serious risks to public safety and security. Existing evaluations often fail to…
Certifying Counterfactual Bias in LLMs
Isha Chaudhary, Qian Hu, Manoj Kumar +3
Large Language Models (LLMs) can produce biased responses that can cause representational harms. However, conventional studies are insufficient to thoroughly evaluate biases across…