97 citations · 209 across the 25 of their papers we have counts for
27 papers · 1 filter
Goal Alignment in LLM-Based User Simulators for Conversational AI
Shuhaib Mehri, Xiaocheng Yang, Takyoung Kim +3
User simulators are essential to conversational AI, enabling scalable agent development and evaluation through simulated interactions. While current Large Language Models (LLMs) ha…
LMUnit: Fine-grained Evaluation with Natural Language Unit Tests
Jon Saad-Falcon, Rajan Vivek, William Berrios +6
As language models become integral to critical workflows, assessing their behavior remains a fundamental challenge -- human evaluation is costly and noisy, while automated metrics…
CESAR: Automatic Induction of Compositional Instructions for Multi-turn Dialogs
Taha Aksu, Devamanyu Hazarika, Shikib Mehri +4
Instruction-based multitasking has played a critical role in the success of large language models (LLMs) in multi-turn dialog applications. While publicly available LLMs have shown…
Data-Efficient Alignment of Large Language Models with Human Feedback Through Natural Language
Di Jin, Shikib Mehri, Devamanyu Hazarika +4
Learning from human feedback is a prominent technique to align the output of large language models (LLMs) with human expectations. Reinforcement learning from human feedback (RLHF)…
Understanding the Effectiveness of Very Large Language Models on Dialog Evaluation
Jessica Huynh, Cathy Jiao, Prakhar Gupta +4
Language models have steadily increased in size over the past few years. They achieve a high level of performance on various natural language processing (NLP) tasks such as questio…
Interactive Evaluation of Dialog Track at DSTC9
Shikib Mehri, Yulan Feng, Carla Gordon +3
The ultimate goal of dialog research is to develop systems that can be effectively used in interactive settings by real users. To this end, we introduced the Interactive Evaluation…
LAD: Language Models as Data for Zero-Shot Dialog
Shikib Mehri, Yasemin Altun, Maxine Eskenazi
To facilitate zero-shot generalization in taskoriented dialog, this paper proposes Language Models as Data (LAD). LAD is a paradigm for creating diverse and accurate synthetic data…
InstructDial: Improving Zero and Few-shot Generalization in Dialogue through Instruction Tuning
Prakhar Gupta, Cathy Jiao, Yi-Ting Yeh +3
Instruction tuning is an emergent paradigm in NLP wherein natural language instructions are leveraged with language models to induce zero-shot performance on unseen tasks. Instruct…
Report from the NSF Future Directions Workshop on Automatic Evaluation of Dialog: Research Directions and Challenges
Shikib Mehri, Jinho Choi, Luis Fernando D'Haro +13
This is a report on the NSF Future Directions Workshop on Automatic Evaluation of Dialog. The workshop explored the current state of the art along with its limitations and suggeste…
A Comprehensive Assessment of Dialog Evaluation Metrics
Yi-Ting Yeh, Maxine Eskenazi, Shikib Mehri
Automatic evaluation metrics are a crucial component of dialog systems research. Standard language evaluation metrics are known to be ineffective for evaluating dialog. As such, re…
Schema-Guided Paradigm for Zero-Shot Dialog
Shikib Mehri, Maxine Eskenazi
Developing mechanisms that flexibly adapt dialog systems to unseen tasks and domains is a major challenge in dialog research. Neural models implicitly memorize task-specific dialog…
GenSF: Simultaneous Adaptation of Generative Pre-trained Models and Slot Filling
Shikib Mehri, Maxine Eskenazi
In transfer learning, it is imperative to achieve strong alignment between a pre-trained model and a downstream task. Prior work has done this by proposing task-specific pre-traini…
Overview of the Ninth Dialog System Technology Challenge: DSTC9
Chulaka Gunasekara, Seokhwan Kim, Luis Fernando D'Haro +36
This paper introduces the Ninth Dialog System Technology Challenge (DSTC-9). This edition of the DSTC focuses on applying end-to-end dialog technologies for four distinct tasks in…
Reasoning Over History: Context Aware Visual Dialog
Muhammad A. Shah, Shikib Mehri, Tejas Srinivasan
While neural models have been shown to exhibit strong performance on single-turn visual question answering (VQA) tasks, extending VQA to a multi-turn, conversational setting remain…
STAR: A Schema-Guided Dialog Dataset for Transfer Learning
Johannes E. M. Mosig, Shikib Mehri, Thomas Kober
We present STAR, a schema-guided task-oriented dialog dataset consisting of 127,833 utterances and knowledge base queries across 5,820 task-oriented dialogs in 13 domains that is e…
Example-Driven Intent Prediction with Observers
Shikib Mehri, Mihail Eric
A key challenge of dialog systems research is to effectively and efficiently adapt to new domains. A scalable paradigm for adaptation necessitates the development of generalizable…
DialoGLUE: A Natural Language Understanding Benchmark for Task-Oriented Dialogue
Shikib Mehri, Mihail Eric, Dilek Hakkani-Tur
A long-standing goal of task-oriented dialogue research is the ability to flexibly adapt dialogue models to new domains. To progress research in this direction, we introduce DialoG…
Unsupervised Evaluation of Interactive Dialog with DialoGPT
Shikib Mehri, Maxine Eskenazi
It is important to define meaningful and interpretable automatic evaluation metrics for open-domain dialog research. Standard language generation metrics have been shown to be inef…
"None of the Above":Measure Uncertainty in Dialog Response Retrieval
Yulan Feng, Shikib Mehri, Maxine Eskenazi +1
This paper discusses the importance of uncovering uncertainty in end-to-end dialog tasks, and presents our experimental results on uncertainty classification on the Ubuntu Dialog C…
USR: An Unsupervised and Reference Free Evaluation Metric for Dialog Generation
Shikib Mehri, Maxine Eskenazi
The lack of meaningful automatic evaluation metrics for dialog has impeded open-domain dialog research. Standard language generation metrics have been shown to be ineffective for e…
CMU GetGoing: An Understandable and Memorable Dialog System for Seniors
Shikib Mehri, Alan W Black, Maxine Eskenazi
Voice-based technologies are typically developed for the average user, and thus generally not tailored to the specific needs of any subgroup of the population, like seniors. This p…
Multi-Granularity Representations of Dialog
Shikib Mehri, Maxine Eskenazi
Neural models of dialog rely on generalized latent representations of language. This paper introduces a novel training procedure which explicitly learns multiple representations of…
Structured Fusion Networks for Dialog
Shikib Mehri, Tejas Srinivasan, Maxine Eskenazi
Neural dialog models have exhibited strong performance, however their end-to-end nature lacks a representation of the explicit structure of dialog. This results in a loss of genera…
Investigating Evaluation of Open-Domain Dialogue Systems With Human Generated Multiple References
Prakhar Gupta, Shikib Mehri, Tiancheng Zhao +3
The aim of this paper is to mitigate the shortcomings of automatic evaluation of open-domain dialog systems through multi-reference evaluation. Existing metrics have been shown to…