24 citations · 24 across the 5 of their papers we have counts for
4 papers · 1 filter
Probing the Robustness of Trained Metrics for Conversational Dialogue Systems
Jan Deriu, Don Tuggener, Pius von Däniken +1
This paper introduces an adversarial method to stress-test trained metrics to evaluate conversational dialogue systems. The method leverages Reinforcement Learning to find response…
Spot The Bot: A Robust and Efficient Framework for the Evaluation of Conversational Dialogue Systems
Jan Deriu, Don Tuggener, Pius von Däniken +6
The lack of time-efficient and reliable evaluation methods hamper the development of conversational dialogue systems (chatbots). Evaluations requiring humans to converse with chatb…
A Methodology for Creating Question Answering Corpora Using Inverse Data Annotation
Jan Deriu, Katsiaryna Mlynchyk, Philippe Schläpfer +6
In this paper, we introduce a novel methodology to efficiently construct a corpus for question answering over structured data. For this, we introduce an intermediate representation…
Towards a Metric for Automated Conversational Dialogue System Evaluation and Improvement
Jan Deriu, Mark Cieliebak
We present "AutoJudge", an automated evaluation method for conversational dialogue systems. The method works by first generating dialogues based on self-talk, i.e. dialogue systems…