14 papers
Beyond Static Leaderboards: Predictive Validity for the Evaluation of LLM Agents
Dhaval C. Patel, Kaoutar El Maghraoui, Shuxin Lin +58
Agent benchmarks are growing fast, but no single benchmark touches more than four or five of the dimensions that deployment exposes. This paper aggregates the largest coordinated d…
Beyond Final Answers: Auditing Trajectory-Level Hallucinations in Multi-Agent Industrial Workflows
Harshada Badave, Santosh Borse, Andrea Gomez +6
Large Language Models (LLMs) are increasingly deployed as autonomous agents that reason, use tools, and act over multiple steps. Yet most hallucination benchmarks still evaluate on…
Code-Guided Reasoning for Small Language Models: Evaluating Executable MCQA Scaffolds
Prateek Biswas, Dhaval Patel, Vedant Khandelwal +2
Multiple-choice QA benchmarks usually evaluate small language models (SLMs) as direct answerers, but deployed language-model systems increasingly rely on external scaffolds such as…
DiagnosticIQ: A Benchmark for LLM-Based Industrial Maintenance Action Recommendation from Symbolic Rules
Devin Yasith De Silva, Dhaval Patel, Christodoulos Constantinides +7
Monitoring complex industrial assets relies on engineer-authored symbolic rules that trigger based on sensor conditions and prompt technicians to perform corrective actions. The bo…
Results and Retrospective Analysis of the CODS 2025 AssetOpsBench Challenge
Dhaval Patel, Chathurangi Shyalika, Suryanarayana Reddy Yarrabothula +4
Competition retrospectives are useful when they explain what a leaderboard measured, how hidden evaluation changed conclusions, and which design patterns were rewarded. We revisit…
PHMForge: Evaluating LLM Agents on Industrial Prognostics through MCP-Native, Algorithm-Grounded Tools
Tianjun Feng, Yunfeng Chen, Chun-Yi Tsai +5
LLM agents are beginning to invoke industrial asset-management tools through the Model Context Protocol (MCP), yet whether they can act reliably on this substrate for safety-critic…