14 papers
From Portraits to Pathology: Probing Diagnostic Overreach in Vision-Language Models
Karan Dua, Amit Agarwal, Hitesh Laxmichand Patel +8
Many medical conditions require clinical assessment of symptoms, history, and functioning rather than judgments from appearance. We study whether vision-language models (VLMs) abst…
Do Image-Text Metrics Respect Semantic Invariances?
Amit Agarwal, Hitesh Laxmichand Patel, Meizhu Liu +9
Reference-free image-to-text evaluators are now standard for scoring image-caption alignment, yet it is unclear whether they respect semantic invariances. We present an invariance…
Robust Audio-Text Retrieval via Cross-Modal Attention and Hybrid Loss
Meizhu Liu, Matthew Rowe, Amit Agarwal +8
Audio-text retrieval enables semantic alignment between audio content and natural language queries, supporting applications in multimedia search, accessibility, and surveillance. H…
SPENCE: A Syntactic Probe for Detecting Contamination in NL2SQL Benchmarks
Mohammadtaher Safarzadeh, Hitesh Laxmichand Patel, Afshin Orojlooyjadid +2
Large language models (LLMs) have achieved strong performance on natural language to SQL (NL2SQL) benchmarks, yet their reported accuracy may be inflated by contamination from benc…
When Vision-Language Models Judge Without Seeing: Exposing Informativeness Bias
Xiaohan Zou, Roshan Sridhar, Mohammadtaher Safarzadeh +1
The reliability of VLM-as-a-Judge is critical for the automatic evaluation of vision-language models (VLMs). Despite recent progress, our analysis reveals that VLM-as-a-Judge often…
PAR-RAG: Planned Active Retrieval and Reasoning for Multi-Hop Question Answering
Xingyu Li, Rongguang Wang, Yuying Wang +5
Large language models (LLMs) remain brittle on multi-hop question answering (MHQA), where answering requires combining evidence across documents through retrieval and reasoning. It…
LLM NL2SQL Robustness: Surface Noise vs. Linguistic Variation in Traditional and Agentic Settings
Lifu Tu, Rongguang Wang, Tao Sheng +2
Robustness evaluation for Natural Language to SQL (NL2SQL) systems is essential because real-world database environments are dynamic, noisy, and continuously evolving, whereas conv…
Chart-RL: Generalized Chart Comprehension via Reinforcement Learning with Verifiable Rewards
Xin Zhang, Xingyu Li, Rongguang Wang +4
Accurate chart comprehension represents a critical challenge in advancing multimodal learning systems, as extensive information is compressed into structured visual representations…
GraphER: An Efficient Graph-Based Enrichment and Reranking Method for Retrieval-Augmented Generation
Ruizhong Miao, Yuying Wang, Rongguang Wang +4
Semantic search in retrieval-augmented generation (RAG) systems is often insufficient for complex information needs, particularly when relevant evidence is scattered across multipl…
OraPlan-SQL: A Planning-Centric Framework for Complex Bilingual NL2SQL Reasoning
Marianne Menglin Liu, Sai Ashish Somayajula, Syed Fahad Allam Shah +2
We present OraPlan-SQL, our system for the Archer NL2SQL Evaluation Challenge 2025, a bilingual benchmark requiring complex reasoning such as arithmetic, commonsense, and hypotheti…
Can LLMs Narrate Tabular Data? An Evaluation Framework for Natural Language Representations of Text-to-SQL System Outputs
Jyotika Singh, Weiyi Sun, Amit Agarwal +4
In modern industry systems like multi-turn chat agents, Text-to-SQL technology bridges natural language (NL) questions and database (DB) querying. The conversion of tabular DB resu…
Aligning LLMs for Multilingual Consistency in Enterprise Applications
Amit Agarwal, Hansa Meghwani, Hitesh Laxmichand Patel +3
Large language models (LLMs) remain unreliable for global enterprise applications due to substantial performance gaps between high-resource and mid/low-resource languages, driven b…
PCRI: Measuring Context Robustness in Multimodal Models for Enterprise Applications
Hitesh Laxmichand Patel, Amit Agarwal, Srikant Panda +6
The reliability of Multimodal Large Language Models (MLLMs) in real-world settings is often undermined by sensitivity to irrelevant or distracting visual context, an aspect not cap…
RCI: A Score for Evaluating Global and Local Reasoning in Multimodal Benchmarks
Amit Agarwal, Hitesh Laxmichand Patel, Srikant Panda +7
Multimodal Large Language Models (MLLMs) have achieved impressive results on vision-language benchmarks, yet it remains unclear whether these benchmarks assess genuine global reaso…