22 papers
VELD-QA: A Large-Scale Dataset for Visual Evidence Grounding and Question Answering in Long Documents
Zhivar Sourati, Marianne Menglin Liu, Rongguang Wang +12
Existing long-document QA benchmarks rarely combine large-scale human-authored questions with detailed visual evidence annotations, limiting diagnosis of evidence localization and…
Same Context, Different Agents: A Behavioral Study of Semantic Layers, Multi-Agent Orchestration and Memory Compaction in Data Agents
Zhivar Sourati, Marianne Menglin Liu, Zheng Wang +6
Answering questions over heterogeneous data means combining evidence from files, databases, documents and the web. Data agents automate this, but their success depends on context.…
From Portraits to Pathology: Probing Diagnostic Overreach in Vision-Language Models
Karan Dua, Amit Agarwal, Hitesh Laxmichand Patel +8
Many medical conditions require clinical assessment of symptoms, history, and functioning rather than judgments from appearance. We study whether vision-language models (VLMs) abst…
PLSQLBench: Benchmarking LLM Systems for Executable Procedural Database Programming
Marianne Menglin Liu, Leonid Boytsov, Daniel W. Peterson +13
We present PLSQLBench, to our knowledge the first benchmark for evaluating whether LLMs can write executable PL/SQL programs, with correctness measured through execution-based test…
SOMA-SQL: Resolving Multi-Source Ambiguity in NL-to-SQL via Synthetic Log and Execution Probing
Sai Ashish Somayajula, Marianne Menglin Liu, Chuan Lei +9
Natural language interfaces to databases aim to translate user questions into executable SQL, yet remain brittle in real-world settings where questions are underspecified and schem…
MT-OSC: Path for LLMs that Get Lost in Multi-Turn Conversation
Jyotika Singh, Fang Tu, Miguel Ballesteros +6
Large language models (LLMs) suffer significant performance degradation when user instructions and context are distributed over multiple conversational turns, yet multi-turn (MT) i…
GSM-SEM: Benchmark and Framework for Generating Semantically Variant Augmentations
Jyotika Singh, Fang Tu, Aziza Mirsaidova +11
Benchmarks like GSM8K are popular measures of mathematical reasoning, but leaderboard gains can overstate true capability due to memorization of fixed test sets. Most robustness va…
Do Image-Text Metrics Respect Semantic Invariances?
Amit Agarwal, Hitesh Laxmichand Patel, Meizhu Liu +9
Reference-free image-to-text evaluators are now standard for scoring image-caption alignment, yet it is unclear whether they respect semantic invariances. We present an invariance…
Robust Audio-Text Retrieval via Cross-Modal Attention and Hybrid Loss
Meizhu Liu, Matthew Rowe, Amit Agarwal +8
Audio-text retrieval enables semantic alignment between audio content and natural language queries, supporting applications in multimedia search, accessibility, and surveillance. H…
JTPRO: A Joint Tool-Prompt Reflective Optimization Framework for Language Agents
Sandip Ghoshal, Anshul Mittal, Jyotika Singh +9
Large language model (LLM) agents augmented with external tools often struggle as number of tools grow large and become domain-specific. In such settings, ambiguous tool descriptio…
DiffuMask: Diffusion Language Model for Token-level Prompt Pruning
Caleb Zheng, Jyotika Singh, Fang Tu +6
In-Context Learning and Chain-of-Thought prompting improve reasoning in large language models (LLMs). These typically come at the cost of longer, more expensive prompts that may co…
PAR-RAG: Planned Active Retrieval and Reasoning for Multi-Hop Question Answering
Xingyu Li, Rongguang Wang, Yuying Wang +5
Large language models (LLMs) remain brittle on multi-hop question answering (MHQA), where answering requires combining evidence across documents through retrieval and reasoning. It…
Think Twice Before You Write -- an Entropy-based Decoding Strategy to Enhance LLM Reasoning
Jiashu He, Meizhu Liu, Olaitan P Olaleye +9
Decoding strategies play a central role in shaping the reasoning ability of large language models (LLMs). Traditional methods such as greedy decoding and beam search often suffer f…
GraphER: An Efficient Graph-Based Enrichment and Reranking Method for Retrieval-Augmented Generation
Ruizhong Miao, Yuying Wang, Rongguang Wang +4
Semantic search in retrieval-augmented generation (RAG) systems is often insufficient for complex information needs, particularly when relevant evidence is scattered across multipl…
Open Agent Specification (Agent Spec): A Unified Representation for AI Agents
Soufiane Amini, Yassine Benajiba, Cesare Bernardis +16
The proliferation of agent frameworks has led to fragmentation in how agents are defined, executed, and evaluated. Existing systems differ in their abstractions, data flow semantic…
Routesplain: Towards Faithful and Intervenable Routing for Software-related Tasks
Adam Štorek, Vikas Upadhyay, Marianne Menglin Liu +6
LLMs now tackle a wide range of software-related tasks, yet we show that their performance varies markedly both across and within these tasks. Routing user queries to the appropria…
OraPlan-SQL: A Planning-Centric Framework for Complex Bilingual NL2SQL Reasoning
Marianne Menglin Liu, Sai Ashish Somayajula, Syed Fahad Allam Shah +2
We present OraPlan-SQL, our system for the Archer NL2SQL Evaluation Challenge 2025, a bilingual benchmark requiring complex reasoning such as arithmetic, commonsense, and hypotheti…
Can LLMs Narrate Tabular Data? An Evaluation Framework for Natural Language Representations of Text-to-SQL System Outputs
Jyotika Singh, Weiyi Sun, Amit Agarwal +4
In modern industry systems like multi-turn chat agents, Text-to-SQL technology bridges natural language (NL) questions and database (DB) querying. The conversion of tabular DB resu…
Aligning LLMs for Multilingual Consistency in Enterprise Applications
Amit Agarwal, Hansa Meghwani, Hitesh Laxmichand Patel +3
Large language models (LLMs) remain unreliable for global enterprise applications due to substantial performance gaps between high-resource and mid/low-resource languages, driven b…
LAD-RAG: Layout-aware Dynamic RAG for Visually-Rich Document Understanding
Zhivar Sourati, Zheng Wang, Marianne Menglin Liu +8
Question answering over visually rich documents (VRDs) requires reasoning not only over isolated content but also over documents' structural organization and cross-page dependencie…
PCRI: Measuring Context Robustness in Multimodal Models for Enterprise Applications
Hitesh Laxmichand Patel, Amit Agarwal, Srikant Panda +6
The reliability of Multimodal Large Language Models (MLLMs) in real-world settings is often undermined by sensitivity to irrelevant or distracting visual context, an aspect not cap…
RCI: A Score for Evaluating Global and Local Reasoning in Multimodal Benchmarks
Amit Agarwal, Hitesh Laxmichand Patel, Srikant Panda +7
Multimodal Large Language Models (MLLMs) have achieved impressive results on vision-language benchmarks, yet it remains unclear whether these benchmarks assess genuine global reaso…