Publications (11)
WHISMA: A Speech-LLM to Perform Zero-shot Spoken Language Understanding
Mohan Li, Cong-Thanh Do, Simon Keizer +3
Speech large language models (speech-LLMs) integrate speech and text-based foundation models to provide a unified framework for handling a wide range of downstream tasks. In this p…
Evaluating Large Language Models for Document-grounded Response Generation in Information-Seeking Dialogues
Norbert Braunschweiler, Rama Doddipatla, Simon Keizer +1
In this paper, we investigate the use of large language models (LLMs) like ChatGPT for document-grounded response generation in the context of information-seeking dialogues. For ev…
User Evaluation of a Multi-dimensional Statistical Dialogue System
Simon Keizer, OndÅej DuÅ¡ek, Xingkun Liu +1
We present the first complete spoken dialogue system driven by a multi-dimensional statistical dialogue manager. This framework has been shown to substantially reduce data needs by…
Action State Update Approach to Dialogue Management
Svetlana Stoyanchev, Simon Keizer, Rama Doddipatla
Utterance interpretation is one of the main functions of a dialogue manager, which is the key component of a dialogue system. We propose the action state update approach (ASU) for…
Adversarial learning of neural user simulators for dialogue policy optimisation
Simon Keizer, Caroline Dockes, Norbert Braunschweiler +2
Reinforcement learning based dialogue policies are typically trained in interaction with a user simulator. To obtain an effective and robust policy, this simulator should generate…
Strategic Dialogue Management via Deep Reinforcement Learning
Heriberto Cuayáhuitl, Simon Keizer, Oliver Lemon
Artificially intelligent agents equipped with strategic skills that can negotiate during their interactions with other natural or artificial agents are still underdeveloped. This p…
Conditional Multi-Stage Failure Recovery for Embodied Agents
Youmna Farag, Svetlana Stoyanchev, Mohan Li +2
Embodied agents performing complex tasks are susceptible to execution failures, motivating the need for effective failure recovery mechanisms. In this work, we introduce a conditio…
Towards Learning Transferable Conversational Skills using Multi-dimensional Dialogue Modelling
Simon Keizer, Verena Rieser
Recent statistical approaches have improved the robustness and scalability of spoken dialogue systems. However, despite recent progress in domain adaptation, their reliance on in-d…
Dialogue Strategy Adaptation to New Action Sets Using Multi-dimensional Modelling
Simon Keizer, Norbert Braunschweiler, Svetlana Stoyanchev +1
A major bottleneck for building statistical spoken dialogue systems for new domains and applications is the need for large amounts of training data. To address this problem, we ado…
Prompting Whisper for QA-driven Zero-shot End-to-end Spoken Language Understanding
Mohan Li, Simon Keizer, Rama Doddipatla
Zero-shot spoken language understanding (SLU) enables systems to comprehend user utterances in new domains without prior exposure to training data. Recent studies often rely on lar…
A study on cross-corpus speech emotion recognition and data augmentation
Norbert Braunschweiler, Rama Doddipatla, Simon Keizer +1
Models that can handle a wide range of speakers and acoustic conditions are essential in speech emotion recognition (SER). Often, these models tend to show mixed results when prese…