206 citations · 277 across the 18 of their papers we have counts for
19 papers
A Small Model Asks Better: Fine-Tuning Query Clarification for Retrieval-Augmented Generation
Lihong He, Lucian Popa, Marina Danilevsky
Query clarification is essential for retrieval-augmented generation (RAG) systems that face ambiguous user queries. We consider multi-turn grounded conversational scenarios where a…
MTRAG-UN: A Benchmark for Open Challenges in Multi-Turn RAG Conversations
Sara Rosenthal, Yannis Katsis, Vraj Shah +3
We present MTRAG-UN, a benchmark for exploring open challenges in multi-turn retrieval augmented generation, a popular use of large language models. We release a benchmark of 666 t…
A Longitudinal Study on Different Annotator Feedback Loops in Complex RAG Tasks
Sara Rosenthal, Maeda Hanafi, Yannis Katsis +2
Grounding conversations in existing passages, known as Retrieval-Augmented Generation (RAG), is an important aspect of Chat-Based Assistants powered by Large Language Models (LLMs)…
RAGAPHENE: A RAG Annotation Platform with Human Enhancements and Edits
Kshitij Fadnis, Sara Rosenthal, Maeda Hanafi +2
Retrieval Augmented Generation (RAG) is an important aspect of conversing with Large Language Models (LLMs) when factually correct information is important. LLMs may provide answer…
A Library of LLM Intrinsics for Retrieval-Augmented Generation
Marina Danilevsky, Kristjan Greenewald, Chulaka Gunasekara +13
In the developer community for large language models (LLMs), there is not yet a clean pattern analogous to a software library, to support very large scale collaboration. Even for t…
MTRAG: A Multi-Turn Conversational Benchmark for Evaluating Retrieval-Augmented Generation Systems
Yannis Katsis, Sara Rosenthal, Kshitij Fadnis +7
Retrieval-augmented generation (RAG) has recently become a very popular task for Large Language Models (LLMs). Evaluating them on multi-turn RAG conversations, where the system is…
Emerging Reliance Behaviors in Human-AI Content Grounded Data Generation: The Role of Cognitive Forcing Functions and Hallucinations
Zahra Ashktorab, Qian Pan, Werner Geyer +5
We investigate the impact of hallucinations and Cognitive Forcing Functions in human-AI collaborative content-grounded data generation, focusing on the use of Large Language Models…
InspectorRAGet: An Introspection Platform for RAG Evaluation
Kshitij Fadnis, Siva Sankalp Patel, Odellia Boni +4
Large Language Models (LLM) have become a popular approach for implementing Retrieval Augmented Generation (RAG) systems, and a significant amount of effort has been spent on build…
Evaluating Robustness of Dialogue Summarization Models in the Presence of Naturally Occurring Variations
Ankita Gupta, Chulaka Gunasekara, Hui Wan +3
Dialogue summarization task involves summarizing long conversations while preserving the most salient information. Real-life dialogues often involve naturally occurring variations…
How Can Context Help? Exploring Joint Retrieval of Passage and Personalized Context
Hui Wan, Hongkang Li, Songtao Lu +2
The integration of external personalized context information into document-grounded conversational systems has significant potential business value, but has not been well-studied.…
Semi-Structured Object Sequence Encoders
Rudra Murthy, Riyaz Bhat, Chulaka Gunasekara +5
In this paper we explore the task of modeling semi-structured object sequences; in particular, we focus our attention on the problem of developing a structure-aware input represent…
PriMeSRL-Eval: A Practical Quality Metric for Semantic Role Labeling Systems Evaluation
Ishan Jindal, Alexandre Rademaker, Khoi-Nguyen Tran +4
Semantic role labeling (SRL) identifies the predicate-argument structure in a sentence. This task is usually accomplished in four steps: predicate identification, predicate sense d…
Label Sleuth: From Unlabeled Text to a Classifier in a Few Hours
Eyal Shnarch, Alon Halfon, Ariel Gera +19
Text classification can be useful in many real-world scenarios, saving a lot of time for end users. However, building a custom classifier typically requires coding skills and ML kn…
Learning to Robustly Aggregate Labeling Functions for Semi-supervised Data Programming
Ayush Maheshwari, Krishnateja Killamsetty, Ganesh Ramakrishnan +3
A critical bottleneck in supervised machine learning is the need for large amounts of labeled data which is expensive and time consuming to obtain. However, it has been shown that…
SemEval-2021 Task 9: Fact Verification and Evidence Finding for Tabular Data in Scientific Documents (SEM-TAB-FACTS)
Nancy X. R. Wang, Diwakar Mahajan, Marina Danilevsky +1
Understanding tables is an important and relevant task that involves understanding table structure as well as being able to compare and contrast information within cells. In this p…
Facilitating Knowledge Sharing from Domain Experts to Data Scientists for Building NLP Models
Soya Park, April Wang, Ban Kawas +3
Data scientists face a steep learning curve in understanding a new domain for which they want to build machine learning (ML) models. While input from domain experts could offer val…
A Survey of the State of Explainable AI for Natural Language Processing
Marina Danilevsky, Kun Qian, Ranit Aharonov +3
Recent years have seen important advances in the quality of state-of-the-art models, but this has come at the expense of models becoming less interpretable. This survey presents an…
Pagination: It's what you say, not how long it takes to say it
Joshua Hailpern, Niranjan Damera Venkata, Marina Danilevsky
Pagination - the process of determining where to break an article across pages in a multi-article layout is a common layout challenge for most commercially printed newspapers and m…
KERT: Automatic Extraction and Ranking of Topical Keyphrases from Content-Representative Document Titles
Marina Danilevsky, Chi Wang, Nihit Desai +2
We introduce KERT (Keyphrase Extraction and Ranking by Topic), a framework for topical keyphrase generation and ranking. By shifting from the unigram-centric traditional methods of…