22 papers
A Trust Layer for Agent Evaluation
Mohammadreza Sediqin, Shivali Dalmia, Srinivasa Karthikeya Reddy Kovvuri +1
Deterministic benchmark scores show that an agent received credit, but not whether that credit was earned, reported honestly, or would hold on a second run. We introduce a Trust La…
Same Output, Different Gold: Measuring How Reference Choice Moves a Multilingual Benchmark Score
Parth Kulshreshtha, Shivali Dalmia, Abhishek Mukherji
A benchmark score compares a system output against a reference, and methodological attention falls almost entirely on the first term. We measure the second. The retained annotation…
CuBEs: Culturally-Situated Behavioral Evaluations and the Limitations of Culture-Blind LLM Judges
Hoda Ayad, Tanu Mitra, Abhishek Mukherji
Evaluating the occurrence and triggers of large language model (LLM) behaviors - such as sycophancy, self-preference, or over-confidence - is critical for predicting real-world mod…
SEABED: SouthEast Asian Benchmark for Evaluating Audio Reasoning
Harshit Rajgarhia, Asif Shaik, Rachuri Lokesh +2
Modern audio-language models are no longer judged only on what words they can transcribe, but on whether they can reason over what they hear: recovering meaning that lives in tone…
SAGE: Governed Artifact Generation from Enterprise Guidelines
Mohammadreza Sediqin, Shivali Dalmia, Sumukha Thoppanahalli +2
Enterprise guideline documents mix narrative text, complex tables, and embedded images, and converting them into structured work artifacts still takes two to three days of manual e…
GVD: Governed Versioning and Deduplication for Document Repositories
Mohammadreza Sediqin, Shivali Dalmia, Sumukha Thoppanahalli +1
Document repositories evolve continuously. Guidelines and policies are revised, superseded, and re-uploaded, so the same content recurs in different wording and newer versions refi…
Voice or Stereotype? Disentangling Acoustic and Content-Based Gender in Speech-to-Speech Models
Xiaoqun Liu, Tanu Mitra, Harshit Rajgarhia +1
Speech-to-speech (S2S) models now run inside dubbing, translation, and voice agents. Unlike text models, they hear the speaker's voice, which carries the speaker's gender. A faithf…
RL-ADA: A World-Feedback Framework for Adversarially Robust Enterprise Dialogue Agents
Ram Narayanan, Harshit Rajgarhia, Abhishek Mukherji
Deploying task-oriented dialogue agents in enterprise customer support faces a persistent annotation bottleneck: robust training requires labelled interaction data at scale, yet en…
GUIDE: Governed Unified Intelligence for Document-to-Artifact Generation in Enterprise Settings
Shivali Dalmia, Sumukha Thoppanahalli, Mohammadreza Sediqin +1
Enterprise guideline documents are heterogeneous and multimodal, combining narrative text, complex tables, and embedded images. Existing LLM and VLM systems face hallucinated conte…
DuplexWorld: Can voice agents help you get through the day?
Aryan Vijay Bhosale, Harshit Rajgarhia, Akhil Pothanapalli +3
Speech-to-speech (S2S) voice agents are increasingly being incorporated into enterprise for customer care and as daily companions for consumers owing to the ease of the conversatio…
TORUS: A Unified Closed-Loop Benchmark for Unified Audio Models
Aryan Vijay Bhosale, Harshit Rajgarhia, Abhishek Mukherji +1
Unified audio models capable of audio understanding, audio generation and, increasingly, audio editing are proliferating rapidly. However, each capability is evaluated in isolation…
CaReCoS: A Spectrogram based Visual Benchmark for Cardiac, Respiratory and Cough Sounds
Harshit Rajgarhia, Shuubham Ojha, Akhil Pothanapalli +4
Medical acoustic signals such as respiratory sounds, cardiac auscultations, and cough audio carry rich diagnostic information, yet no existing benchmark evaluates multimodal reason…
World Feedback for Clinical Agents: Diagnosing RL in FHIR Environments
Ananya Mantravadi, Harshit Rajgarhia, Prasanna Desikan +1
Clinical protocol-execution tasks -- checking a lab value, applying a threshold, placing a correctly structured FHIR order -- are natural candidates for RL from world feedback: onc…
Beyond Next-Token Prediction: An RLVR Proof of Concept for Tool-Use Agents on Atlassian Workflows
Karthikeya Aditya Vissa, Sankalp Mane, Ananya Mantravadi +2
Large language models are trained to predict the next token, not to act inside a specific API. In niche enterprise SaaS workflows -- where success means hitting the right endpoint…
MedMosaic: A Challenging Large Scale Benchmark of Diverse Medical Audio
Harshit Rajgarhia, Shuubham Ojha, Asif Shaik +4
Medical audio data is difficult to collect due to privacy regulations and high annotation costs arising from domain expertise. Thus, existing benchmarks tend to underrepresent comp…
CareTransition-Audit: A Benchmark to Audit Discharge Summaries for Efficient Care Transitions
Akshat Dasula, Prasanna Desikan, Jaideep Srivastava +2
Incomplete or inconsistent discharge documentation drives care fragmentation and avoidable readmissions. Despite its critical role in patient safety, auditing discharge summaries r…
ART: Action-based Reasoning Task Benchmarking for Medical AI Agents
Ananya Mantravadi, Shivali Dalmia, Abhishek Mukherji
Reliable clinical decision support requires medical AI agents capable of safe, multi-step reasoning over structured electronic health records (EHRs). While large language models (L…
LegalWiz: A Multi-Agent Generation Framework for Contradiction Detection in Legal Documents
Ananya Mantravadi, Shivali Dalmia, Olga Pospelova +3
Retrieval-Augmented Generation (RAG) integrates large language models (LLMs) with external sources, but unresolved contradictions in retrieved evidence often lead to hallucinations…
Scalable multilingual PII annotation for responsible AI in LLMs
Bharti Meena, Joanna Skubisz, Harshit Rajgarhia +5
As Large Language Models (LLMs) gain wider adoption, ensuring their reliable handling of Personally Identifiable Information (PII) across diverse regulatory contexts has become ess…
An Evaluation Study of Hybrid Methods for Multilingual PII Detection
Harshit Rajgarhia, Suryam Gupta, Asif Shaik +4
The detection of Personally Identifiable Information (PII) is critical for privacy compliance but remains challenging in low-resource languages due to linguistic diversity and limi…
GAZE:Governance-Aware pre-annotation for Zero-shot World Model Environments
Leela Krishna, Mengyang Zhao, Saicharithreddy Pasula +2
Training robust world models requires large-scale, precisely labeled multimodal datasets, a process historically bottlenecked by slow and expensive manual annotation. We present a…
Human + AI for Accelerating Ad Localization Evaluation
Harshit Rajgarhia, Shivali Dalmia, Mengyang Zhao +2
Adapting advertisements for multilingual audiences requires more than simple text translation; it demands preservation of visual consistency, spatial alignment, and stylistic integ…