11 papers · 2 filters
Synthetic Scenario Generation for Evaluation of Industry 4.0 Agents
Sagar Chethan Kumar, Rohith Kanathur, Dhaval Patel +1
Industrial agent benchmarks require realistic evaluation scenarios that integrate telemetry, failure modes, maintenance records, and domain standards. However, existing benchmarks…
Beyond Static Leaderboards: Predictive Validity for the Evaluation of LLM Agents
Dhaval C. Patel, Kaoutar El Maghraoui, Shuxin Lin +58
Agent benchmarks are growing fast, but no single benchmark touches more than four or five of the dimensions that deployment exposes. This paper aggregates the largest coordinated d…
EvoFlux: Inference-Time Evolution of Executable Tool Workflows for Compact Agents
Kushal Raj Bhandari, Ling Yue, Ching-Yun Ko +4
Small language models (1.5B-4B parameters) offer low-cost, low-latency tool use but often produce plausible plans that fail during execution due to unavailable tools, missing argum…
Beyond Final Answers: Auditing Trajectory-Level Hallucinations in Multi-Agent Industrial Workflows
Harshada Badave, Santosh Borse, Andrea Gomez +6
Large Language Models (LLMs) are increasingly deployed as autonomous agents that reason, use tools, and act over multiple steps. Yet most hallucination benchmarks still evaluate on…
Evaluating Temporal Semantic Caching and Workflow Optimization in Agentic Plan-Execute Pipelines
Alimurtaza Mustafa Merchant, Krish Veera, Sajal Kumar Goyla +3
Industrial asset operations workflows are latency-sensitive because a single user query may require coordination over sensor data, work orders, failure modes, forecasting tools, an…
SPIN: Structural LLM Planning via Iterative Navigation for Industrial Tasks
Yusuke Ozaki, Dhaval Patel
Industrial LLM agent systems often separate planning from execution, yet LLM planners frequently produce structurally invalid or unnecessarily long workflows, leading to brittle fa…
MCP-Cosmos: World Model-Augmented Agents for Complex Task Execution in MCP Environments
Giridhar Ganapavarapu, Dhaval Patel
The Model Context Protocol (MCP) has unified the interface between Large Language Models (LLMs) and external tools, yet a fundamental gap remains in how agents conceptualize the en…
DiagnosticIQ: A Benchmark for LLM-Based Industrial Maintenance Action Recommendation from Symbolic Rules
Devin Yasith De Silva, Dhaval Patel, Christodoulos Constantinides +8
Monitoring complex industrial assets relies on engineer-authored symbolic rules that trigger on sensor conditions and prompt technicians to perform corrective actions. The bottlene…
Results and Retrospective Analysis of the CODS 2025 AssetOpsBench Challenge
Dhaval Patel, Chathurangi Shyalika, Suryanarayana Reddy Yarrabothula +4
Competition retrospectives are useful when they explain what a leaderboard measured, how hidden evaluation changed conclusions, and which design patterns were rewarded. We revisit…
IndustryAssetEQA: A Neurosymbolic Operational Intelligence System for Embodied Question Answering in Industrial Asset Maintenance
Chathurangi Shyalika, Dhaval Patel, Amit Sheth
Industrial maintenance environments increasingly rely on AI systems to assist operators in understanding asset behavior, diagnosing failures, and evaluating interventions. Although…
PHMForge: Evaluating LLM Agents on Industrial Prognostics through MCP-Native, Algorithm-Grounded Tools
Yusheng Li, Tianjun Feng, Yunfeng Chen +6
LLM agents are beginning to invoke industrial asset-management tools through the Model Context Protocol (MCP), yet whether they can act reliably on this substrate for safety-critic…