164 citations · 515 across the 28 of their papers we have counts for
30 papers · 1 filter
Understanding the Effectiveness of Very Large Language Models on Dialog Evaluation
Jessica Huynh, Cathy Jiao, Prakhar Gupta +4
Language models have steadily increased in size over the past few years. They achieve a high level of performance on various natural language processing (NLP) tasks such as questio…
Interactive Evaluation of Dialog Track at DSTC9
Shikib Mehri, Yulan Feng, Carla Gordon +3
The ultimate goal of dialog research is to develop systems that can be effectively used in interactive settings by real users. To this end, we introduced the Interactive Evaluation…
LAD: Language Models as Data for Zero-Shot Dialog
Shikib Mehri, Yasemin Altun, Maxine Eskenazi
To facilitate zero-shot generalization in taskoriented dialog, this paper proposes Language Models as Data (LAD). LAD is a paradigm for creating diverse and accurate synthetic data…
DialCrowd 2.0: A Quality-Focused Dialog System Crowdsourcing Toolkit
Jessica Huynh, Ting-Rui Chiang, Jeffrey Bigham +1
Dialog system developers need high-quality data to train, fine-tune and assess their systems. They often use crowdsourcing for this since it provides large quantities of data from…
InstructDial: Improving Zero and Few-shot Generalization in Dialogue through Instruction Tuning
Prakhar Gupta, Cathy Jiao, Yi-Ting Yeh +3
Instruction tuning is an emergent paradigm in NLP wherein natural language instructions are leveraged with language models to induce zero-shot performance on unseen tasks. Instruct…
Report from the NSF Future Directions Workshop on Automatic Evaluation of Dialog: Research Directions and Challenges
Shikib Mehri, Jinho Choi, Luis Fernando D'Haro +13
This is a report on the NSF Future Directions Workshop on Automatic Evaluation of Dialog. The workshop explored the current state of the art along with its limitations and suggeste…
A Survey of NLP-Related Crowdsourcing HITs: what works and what does not
Jessica Huynh, Jeffrey Bigham, Maxine Eskenazi
Crowdsourcing requesters on Amazon Mechanical Turk (AMT) have raised questions about the reliability of the workers. The AMT workforce is very diverse and it is not possible to mak…
A Comprehensive Assessment of Dialog Evaluation Metrics
Yi-Ting Yeh, Maxine Eskenazi, Shikib Mehri
Automatic evaluation metrics are a crucial component of dialog systems research. Standard language evaluation metrics are known to be ineffective for evaluating dialog. As such, re…
Schema-Guided Paradigm for Zero-Shot Dialog
Shikib Mehri, Maxine Eskenazi
Developing mechanisms that flexibly adapt dialog systems to unseen tasks and domains is a major challenge in dialog research. Neural models implicitly memorize task-specific dialog…
GenSF: Simultaneous Adaptation of Generative Pre-trained Models and Slot Filling
Shikib Mehri, Maxine Eskenazi
In transfer learning, it is imperative to achieve strong alignment between a pre-trained model and a downstream task. Prior work has done this by proposing task-specific pre-traini…
Overview of the Ninth Dialog System Technology Challenge: DSTC9
Chulaka Gunasekara, Seokhwan Kim, Luis Fernando D'Haro +36
This paper introduces the Ninth Dialog System Technology Challenge (DSTC-9). This edition of the DSTC focuses on applying end-to-end dialog technologies for four distinct tasks in…
Unsupervised Evaluation of Interactive Dialog with DialoGPT
Shikib Mehri, Maxine Eskenazi
It is important to define meaningful and interpretable automatic evaluation metrics for open-domain dialog research. Standard language generation metrics have been shown to be inef…
Report from the NSF Future Directions Workshop, Toward User-Oriented Agents: Research Directions and Challenges
Maxine Eskenazi, Tiancheng Zhao
This USER Workshop was convened with the goal of defining future research directions for the burgeoning intelligent agent research community and to communicate them to the National…
"None of the Above":Measure Uncertainty in Dialog Response Retrieval
Yulan Feng, Shikib Mehri, Maxine Eskenazi +1
This paper discusses the importance of uncovering uncertainty in end-to-end dialog tasks, and presents our experimental results on uncertainty classification on the Ubuntu Dialog C…
USR: An Unsupervised and Reference Free Evaluation Metric for Dialog Generation
Shikib Mehri, Maxine Eskenazi
The lack of meaningful automatic evaluation metrics for dialog has impeded open-domain dialog research. Standard language generation metrics have been shown to be ineffective for e…
CMU GetGoing: An Understandable and Memorable Dialog System for Seniors
Shikib Mehri, Alan W Black, Maxine Eskenazi
Voice-based technologies are typically developed for the average user, and thus generally not tailored to the specific needs of any subgroup of the population, like seniors. This p…
Multi-Granularity Representations of Dialog
Shikib Mehri, Maxine Eskenazi
Neural models of dialog rely on generalized latent representations of language. This paper introduces a novel training procedure which explicitly learns multiple representations of…
Structured Fusion Networks for Dialog
Shikib Mehri, Tejas Srinivasan, Maxine Eskenazi
Neural dialog models have exhibited strong performance, however their end-to-end nature lacks a representation of the explicit structure of dialog. This results in a loss of genera…
Investigating Evaluation of Open-Domain Dialogue Systems With Human Generated Multiple References
Prakhar Gupta, Shikib Mehri, Tiancheng Zhao +3
The aim of this paper is to mitigate the shortcomings of automatic evaluation of open-domain dialog systems through multi-reference evaluation. Existing metrics have been shown to…
Pretraining Methods for Dialog Context Representation Learning
Shikib Mehri, Evgeniia Razumovskaia, Tiancheng Zhao +1
This paper examines various unsupervised pretraining objectives for learning dialog context representations. Two novel methods of pretraining dialog context encoders are proposed,…
Rethinking Action Spaces for Reinforcement Learning in End-to-end Dialog Agents with Latent Variable Models
Tiancheng Zhao, Kaige Xie, Maxine Eskenazi
Defining action spaces for conversational agents and optimizing their decision-making process with reinforcement learning is an enduring challenge. Common practice has been to use…
Beyond Turing: Intelligent Agents Centered on the User
Maxine Eskenazi, Shikib Mehri, Evgeniia Razumovskaia +1
Most research on intelligent agents centers on the agent and not on the user. We look at the origins of agent-centric research for slot-filling, gaming and chatbot agents. We then…
Context-Aware Dialog Re-Ranking for Task-Oriented Dialog Systems
Junki Ohmura, Maxine Eskenazi
Dialog response ranking is used to rank response candidates by considering their relation to the dialog history. Although researchers have addressed this concept for open-domain di…
Zero-Shot Dialog Generation with Cross-Domain Latent Actions
Tiancheng Zhao, Maxine Eskenazi
This paper introduces zero-shot dialog generation (ZSDG), as a step towards neural dialog systems that can instantly generalize to new situations with minimal data. ZSDG enables an…