4 citations · 13 across the 22 of their papers we have counts for
16 papers · 1 filter
GUI-KV: Efficient GUI Agents via KV Cache with Spatio-Temporal Awareness
Kung-Hsiang Huang, Haoyi Qiu, Yutong Dai +2
Graphical user interface (GUI) agents built on vision-language models have emerged as a promising approach to automate human-computer workflows. However, they also face the ineffic…
Benchmarking Deep Search over Heterogeneous Enterprise Data
Prafulla Kumar Choubey, Xiangyu Peng, Shilpa Bhagavath +3
We present a new benchmark for evaluating Deep Search--a realistic and complex form of retrieval-augmented generation (RAG) that requires source-aware, multi-hop reasoning over div…
CRMArena-Pro: Holistic Assessment of LLM Agents Across Diverse Business Scenarios and Interactions
Kung-Hsiang Huang, Akshara Prabhakar, Onkar Thorat +6
While AI agents hold transformative potential in business, effective performance benchmarking is hindered by the scarcity of public, realistic business data on widely used platform…
BingoGuard: LLM Content Moderation Tools with Risk Levels
Fan Yin, Philippe Laban, Xiangyu Peng +7
Malicious content generated by large language models (LLMs) can pose varying degrees of harm. Although existing LLM-based moderators can detect harmful content, they struggle to as…
Turning Conversations into Workflows: A Framework to Extract and Evaluate Dialog Workflows for Service AI Agents
Prafulla Kumar Choubey, Xiangyu Peng, Shilpa Bhagavath +3
Automated service agents require well-structured workflows to provide consistent and accurate responses to customer queries. However, these workflows are often undocumented, and th…
Unanswerability Evaluation for Retrieval Augmented Generation
Xiangyu Peng, Prafulla Kumar Choubey, Caiming Xiong +1
Existing evaluation frameworks for retrieval-augmented generation (RAG) systems focus on answerable queries, but they overlook the importance of appropriately rejecting unanswerabl…
SiReRAG: Indexing Similar and Related Information for Multihop Reasoning
Nan Zhang, Prafulla Kumar Choubey, Alexander Fabbri +5
Indexing is an important step towards strong performance in retrieval-augmented generation (RAG) systems. However, existing methods organize data based on either semantic similarit…
CRMArena: Understanding the Capacity of LLM Agents to Perform Professional CRM Tasks in Realistic Environments
Kung-Hsiang Huang, Akshara Prabhakar, Sidharth Dhawan +6
Customer Relationship Management (CRM) systems are vital for modern enterprises, providing a foundation for managing customer interactions and data. Integrating AI agents into CRM…
Do RAG Systems Cover What Matters? Evaluating and Optimizing Responses with Sub-Question Coverage
Kaige Xie, Philippe Laban, Prafulla Kumar Choubey +2
Evaluating retrieval-augmented generation (RAG) systems remains challenging, particularly for open-ended questions that lack definitive answers and require coverage of multiple sub…
Scaling Knowledge Graph Construction through Synthetic Data Generation and Distillation
Prafulla Kumar Choubey, Xin Su, Man Luo +9
Document-level knowledge graph (KG) construction faces a fundamental scaling challenge: existing methods either rely on expensive large language models (LLMs), making them economic…
ReGenesis: LLMs can Grow into Reasoning Generalists via Self-Improvement
Xiangyu Peng, Congying Xia, Xinyi Yang +3
Post-training Large Language Models (LLMs) with explicit reasoning trajectories can enhance their reasoning abilities. However, acquiring such high-quality trajectory data typicall…
Shared Imagination: LLMs Hallucinate Alike
Yilun Zhou, Caiming Xiong, Silvio Savarese +1
Despite the recent proliferation of large language models (LLMs), their training recipes -- model architecture, pre-training data and optimization algorithm -- are often very simil…
Summary of a Haystack: A Challenge to Long-Context LLMs and RAG Systems
Philippe Laban, Alexander R. Fabbri, Caiming Xiong +1
LLMs and RAG systems are now capable of handling millions of input tokens or more. However, evaluating the output quality of such systems on long-context tasks remains challenging,…
Granular Change Accuracy: A More Accurate Performance Metric for Dialogue State Tracking
Taha Aksu, Nancy F. Chen
Current metrics for evaluating Dialogue State Tracking (DST) systems exhibit three primary limitations. They: i) erroneously presume a uniform distribution of slots throughout the…
Lexical Repetitions Lead to Rote Learning: Unveiling the Impact of Lexical Overlap in Train and Test Reference Summaries
Prafulla Kumar Choubey, Alexander R. Fabbri, Caiming Xiong +1
Ideal summarization models should generalize to novel summary-worthy content without remembering reference training summaries by rote. However, a single average performance score o…
Are You Sure? Challenging LLMs Leads to Performance Drops in The FlipFlop Experiment
Philippe Laban, Lidiya Murakhovs'ka, Caiming Xiong +1
The interactive nature of Large Language Models (LLMs) theoretically allows models to refine and improve their answers, yet systematic analysis of the multi-turn behavior of LLMs r…