565 citations · 594 across the 14 of their papers we have counts for
12 papers · 1 filter
TOD-ProcBench: Benchmarking Complex Instruction-Following in Task-Oriented Dialogues
Sarik Ghazarian, Abhinav Gullapalli, Swair Shah +4
In real-world task-oriented dialogue (TOD) settings, agents are required to strictly adhere to complex instructions while conducting multi-turn conversations with customers. These…
Overview of Robust and Multilingual Automatic Evaluation Metrics for Open-Domain Dialogue Systems at DSTC 11 Track 4
Mario Rodríguez-Cantelar, Chen Zhang, Chengguang Tang +5
The advent and fast development of neural networks have revolutionized the research on dialogue systems and subsequently have triggered various challenges regarding their automatic…
ACCENT: An Automatic Event Commonsense Evaluation Metric for Open-Domain Dialogue Systems
Sarik Ghazarian, Yijia Shao, Rujun Han +2
Commonsense reasoning is omnipresent in human communications and thus is an important feature for open-domain dialogue systems. However, evaluating commonsense in dialogue systems…
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao +448
Language models demonstrate both quantitative improvement and new qualitative capabilities with increasing scale. Despite their potentially transformative impact, these new capabil…
What is wrong with you?: Leveraging User Sentiment for Automatic Dialog Evaluation
Sarik Ghazarian, Behnam Hedayatnia, Alexandros Papangelis +2
Accurate automatic evaluation metrics for open-domain dialogs are in high demand. Existing model-based metrics for system response evaluation are trained on human annotated data, w…
DEAM: Dialogue Coherence Evaluation using AMR-based Semantic Manipulations
Sarik Ghazarian, Nuan Wen, Aram Galstyan +1
Automatic evaluation metrics are essential for the rapid development of open-domain dialogue systems as they facilitate hyper-parameter tuning and comparison between models. Althou…
User Response and Sentiment Prediction for Automatic Dialogue Evaluation
Sarik Ghazarian, Behnam Hedayatnia, Alexandros Papangelis +2
Automatic evaluation is beneficial for open-domain dialog system development. However, standard word-overlap metrics (BLEU, ROUGE) do not correlate well with human judgements of op…
Plot-guided Adversarial Example Construction for Evaluating Open-domain Story Generation
Sarik Ghazarian, Zixi Liu, Akash SM +3
With the recent advances of open-domain story generation, the lack of reliable automatic evaluation metrics becomes an increasingly imperative issue that hinders the fast developme…
DiSCoL: Toward Engaging Dialogue Systems through Conversational Line Guided Response Generation
Sarik Ghazarian, Zixi Liu, Tuhin Chakrabarty +3
Having engaging and informative conversations with users is the utmost goal for open-domain conversational systems. Recent advances in transformer-based language models and their a…
ParsiNLU: A Suite of Language Understanding Challenges for Persian
Daniel Khashabi, Arman Cohan, Siamak Shakeri +22
Despite the progress made in recent years in addressing natural language understanding (NLU) challenges, the majority of this progress remains to be concentrated on resource-rich l…
Predictive Engagement: An Efficient Metric For Automatic Evaluation of Open-Domain Dialogue Systems
Sarik Ghazarian, Ralph Weischedel, Aram Galstyan +1
User engagement is a critical metric for evaluating the quality of open-domain dialogue systems. Prior work has focused on conversation-level engagement by using heuristically cons…
Better Automatic Evaluation of Open-Domain Dialogue Systems with Contextualized Embeddings
Sarik Ghazarian, Johnny Tian-Zheng Wei, Aram Galstyan +1
Despite advances in open-domain dialogue systems, automatic evaluation of such systems is still a challenging problem. Traditional reference-based metrics such as BLEU are ineffect…