8 citations · 32 across the 36 of their papers we have counts for
36 papers
ConvApparel: A Benchmark Dataset and Validation Framework for User Simulators in Conversational Recommenders
Ofer Meshi, Krisztian Balog, Sally Goldman +5
The promise of LLM-based user simulators to improve conversational AI is hindered by a critical "realism gap," leading to systems that are optimized for simulated interactions, but…
MDSEval: A Meta-Evaluation Benchmark for Multimodal Dialogue Summarization
Yinhong Liu, Jianfeng He, Hang Su +6
Multimodal Dialogue Summarization (MDS) is a critical task with wide-ranging applications. To support the development of effective MDS models, robust automatic evaluation methods a…
Controllable Conversational Theme Detection Track at DSTC 12
Igor Shalyminov, Hang Su, Jake Vincent +5
Conversational analytics has been on the forefront of transformation driven by the advances in Speech and Natural Language Processing techniques. Rapid adoption of Large Language M…
Multilingual Self-Taught Faithfulness Evaluators
Carlo Alfano, Aymen Al Marjani, Zeno Jonke +3
The growing use of large language models (LLMs) has increased the need for automatic evaluation systems, particularly to address the challenge of information hallucination. Althoug…
PAARS: Persona Aligned Agentic Retail Shoppers
Saab Mansour, Leonardo Perelli, Lorenzo Mainetti +2
In e-commerce, behavioral data is collected for decision making which can be costly and slow. Simulation with LLM powered agents is emerging as a promising alternative for represen…
Faithful, Unfaithful or Ambiguous? Multi-Agent Debate with Initial Stance for Summary Evaluation
Mahnaz Koupaee, Jake W. Vincent, Saab Mansour +9
Faithfulness evaluators based on large language models (LLMs) are often fooled by the fluency of the text and struggle with identifying errors in the summaries. We propose an appro…
GLEAN: Active Generalized Category Discovery with Diverse LLM Feedback
Henry Peng Zou, Siffi Singh, Yi Nian +4
Generalized Category Discovery (GCD) is a practical and challenging open-world task that aims to recognize both known and novel categories in unlabeled data using limited labeled d…
MEMERAG: A Multilingual End-to-End Meta-Evaluation Benchmark for Retrieval Augmented Generation
María Andrea Cruz Blandón, Jayasimha Talur, Bruno Charron +3
Automatic evaluation of retrieval augmented generation (RAG) systems relies on fine-grained dimensions like faithfulness and relevance, as judged by expert human annotators. Meta-e…
Hierarchical Multi-field Representations for Two-Stage E-commerce Retrieval
Niklas Freymuth, Dong Liu, Thomas Ricatte +1
Dense retrieval methods typically target unstructured text data represented as flat strings. However, e-commerce catalogs often include structured information across multiple field…
Structured List-Grounded Question Answering
Mujeen Sung, Song Feng, James Gung +3
Document-grounded dialogue systems aim to answer user queries by leveraging external information. Previous studies have mainly focused on handling free-form documents, often overlo…
DFlow: Diverse Dialogue Flow Simulation with Large Language Models
Wanyu Du, Song Feng, James Gung +4
Developing language model-based dialogue agents requires effective data to train models that can follow specific task logic. However, most existing data simulation methods focus on…
FineSurE: Fine-grained Summarization Evaluation using LLMs
Hwanjun Song, Hang Su, Igor Shalyminov +2
Automated evaluation is crucial for streamlining text summarization benchmarking and model development, given the costly and time-consuming nature of human evaluation. Traditional…
CERET: Cost-Effective Extrinsic Refinement for Text Generation
Jason Cai, Hang Su, Monica Sunkara +2
Large Language Models (LLMs) are powerful models for generation tasks, but they may not generate good quality outputs in their first attempt. Apart from model fine-tuning, existing…
TofuEval: Evaluating Hallucinations of LLMs on Topic-Focused Dialogue Summarization
Liyan Tang, Igor Shalyminov, Amy Wing-mei Wong +11
Single document news summarization has seen substantial progress on faithfulness in recent years, driven by research on the evaluation of factual consistency, or hallucinations. We…
Can Your Model Tell a Negation from an Implicature? Unravelling Challenges With Intent Encoders
Yuwei Zhang, Siffi Singh, Sailik Sengupta +4
Conversational systems often rely on embedding models for intent classification and intent clustering tasks. The advent of Large Language Models (LLMs), which enable instructional…
Semi-Supervised Dialogue Abstractive Summarization via High-Quality Pseudolabel Selection
Jianfeng He, Hang Su, Jason Cai +3
Semi-supervised dialogue summarization (SSDS) leverages model-generated summaries to reduce reliance on human-labeled data and improve the performance of summarization models. Whil…
FLAP: Flow-Adhering Planning with Constrained Decoding in LLMs
Shamik Roy, Sailik Sengupta, Daniele Bonadiman +2
Planning is a crucial task for agents in task oriented dialogs (TODs). Human agents typically resolve user issues by following predefined workflows, decomposing workflow steps into…
MAGID: An Automated Pipeline for Generating Synthetic Multi-modal Datasets
Hossein Aboutalebi, Hwanjun Song, Yusheng Xie +7
Development of multimodal interactive systems is hindered by the lack of rich, multimodal (text, images) conversational data, which is needed in large quantities for LLMs. Previous…
Eliciting Better Multilingual Structured Reasoning from LLMs through Code
Bryan Li, Tamer Alkhouli, Daniele Bonadiman +2
The development of large language models (LLM) has shown progress on reasoning, though studies have largely considered either English or simple reasoning tasks. To address this, we…
DeAL: Decoding-time Alignment for Large Language Models
James Y. Huang, Sailik Sengupta, Daniele Bonadiman +6
Large Language Models (LLMs) are nowadays expected to generate content aligned with human preferences. Current work focuses on alignment at model training time, through techniques…
Enhancing Abstractiveness of Summarization Models through Calibrated Distillation
Hwanjun Song, Igor Shalyminov, Hang Su +3
Sequence-level knowledge distillation reduces the size of Seq2Seq models for more efficient abstractive summarization. However, it often leads to a loss of abstractiveness in summa…
User Simulation with Large Language Models for Evaluating Task-Oriented Dialogue
Sam Davidson, Salvatore Romeo, Raphael Shu +4
One of the major impediments to the development of new task-oriented dialogue (TOD) systems is the need for human evaluation at multiple stages and iterations of the development pr…
NatCS: Eliciting Natural Customer Support Dialogues
James Gung, Emily Moeng, Wesley Rose +3
Despite growing interest in applications based on natural customer support conversations, there exist remarkably few publicly available datasets that reflect the expected character…
Intent Induction from Conversations for Task-Oriented Dialogue Track at DSTC 11
James Gung, Raphael Shu, Emily Moeng +6
With increasing demand for and adoption of virtual assistants, recent work has investigated ways to accelerate bot schema design through the automatic induction of intents or the i…