9 papers
VELD-QA: A Large-Scale Dataset for Visual Evidence Grounding and Question Answering in Long Documents
Zhivar Sourati, Marianne Menglin Liu, Rongguang Wang +12
Existing long-document QA benchmarks rarely combine large-scale human-authored questions with detailed visual evidence annotations, limiting diagnosis of evidence localization and…
Same Context, Different Agents: A Behavioral Study of Semantic Layers, Multi-Agent Orchestration and Memory Compaction in Data Agents
Zhivar Sourati, Marianne Menglin Liu, Zheng Wang +6
Answering questions over heterogeneous data means combining evidence from files, databases, documents and the web. Data agents automate this, but their success depends on context.…
PLSQLBench: Benchmarking LLM Systems for Executable Procedural Database Programming
Marianne Menglin Liu, Leonid Boytsov, Daniel W. Peterson +13
We present PLSQLBench, to our knowledge the first benchmark for evaluating whether LLMs can write executable PL/SQL programs, with correctness measured through execution-based test…
SOMA-SQL: Resolving Multi-Source Ambiguity in NL-to-SQL via Synthetic Log and Execution Probing
Sai Ashish Somayajula, Marianne Menglin Liu, Chuan Lei +9
Natural language interfaces to databases aim to translate user questions into executable SQL, yet remain brittle in real-world settings where questions are underspecified and schem…
PAR-RAG: Planned Active Retrieval and Reasoning for Multi-Hop Question Answering
Xingyu Li, Rongguang Wang, Yuying Wang +5
Large language models (LLMs) remain brittle on multi-hop question answering (MHQA), where answering requires combining evidence across documents through retrieval and reasoning. It…
LLM NL2SQL Robustness: Surface Noise vs. Linguistic Variation in Traditional and Agentic Settings
Lifu Tu, Rongguang Wang, Tao Sheng +2
Robustness evaluation for Natural Language to SQL (NL2SQL) systems is essential because real-world database environments are dynamic, noisy, and continuously evolving, whereas conv…
GraphER: An Efficient Graph-Based Enrichment and Reranking Method for Retrieval-Augmented Generation
Ruizhong Miao, Yuying Wang, Rongguang Wang +4
Semantic search in retrieval-augmented generation (RAG) systems is often insufficient for complex information needs, particularly when relevant evidence is scattered across multipl…
Deep Learning Framework Testing via Model Mutation: How Far Are We?
Yanzhou Mu, Rong Wang, Juan Zhai +7
Deep Learning (DL) frameworks are a fundamental component of DL development. Therefore, the detection of DL framework defects is important and challenging. As one of the most widel…
Understanding LLM-Centric Challenges for Deep Learning Frameworks: An Empirical Analysis
Yanzhou Mu, Rong Wang, Juan Zhai +7
Large language models (LLMs) have driven significant progress across a wide range of real-world applications. Realizing such models requires substantial system-level support. Deep…