58 citations · 147 across the 26 of their papers we have counts for
28 papers
When Robots Mishear Us: Mapping the Safety Risks of Voice-Controlled Embodied AI
Sihan Jia, Oliver Lemon
We investigate whether automatic speech recognition (ASR) errors in user input can lead to unsafe outputs from Embodied AI (EAI) models. We find that ASR errors can lead to harmful…
Multi-party Goal Tracking with LLMs: Comparing Pre-training, Fine-tuning, and Prompt Engineering
Angus Addlesee, Weronika Sieińska, Nancie Gunson +3
This paper evaluates the extent to which current Large Language Models (LLMs) can capture task-oriented multi-party conversations (MPCs). We have recorded and transcribed 29 MPCs b…
SimpleMTOD: A Simple Language Model for Multimodal Task-Oriented Dialogue with Symbolic Scene Representation
Bhathiya Hemanthage, Christian Dondrup, Phil Bartie +1
SimpleMTOD is a simple language model which recasts several sub-tasks in multimodal task-oriented dialogues as sequence prediction tasks. SimpleMTOD is built on a large-scale trans…
An Empirical Study on the Generalization Power of Neural Representations Learned via Visual Guessing Games
Alessandro Suglia, Yonatan Bisk, Ioannis Konstas +4
Guessing games are a prototypical instance of the "learning by interacting" paradigm. This work investigates how well an artificial agent can benefit from playing guessing games wh…
Encoding Syntactic Constituency Paths for Frame-Semantic Parsing with Graph Convolutional Networks
Emanuele Bastianelli, Andrea Vanzo, Oliver Lemon
We study the problem of integrating syntactic information from constituency trees into a neural model in Frame-semantic parsing sub-tasks, namely Target Identification (TI), FrameI…
Imagining Grounded Conceptual Representations from Perceptual Information in Situated Guessing Games
Alessandro Suglia, Antonio Vergari, Ioannis Konstas +4
In visual guessing games, a Guesser has to identify a target object in a scene by asking questions to an Oracle. An effective strategy for the players is to learn conceptual repres…
Explainable Representations of the Social State: A Model for Social Human-Robot Interactions
Daniel Hernández García, Yanchao Yu, Weronika Sieińska +4
In this paper, we propose a minimum set of concepts and signals needed to track the social state during Human-Robot Interaction. We look into the problem of complex continuous inte…
CompGuessWhat?!: A Multi-task Evaluation Framework for Grounded Language Learning
Alessandro Suglia, Ioannis Konstas, Andrea Vanzo +4
Approaches to Grounded Language Learning typically focus on a single task-based final performance measure that may not depend on desirable properties of the learned hidden represen…
Data-Efficient Goal-Oriented Conversation with Dialogue Knowledge Transfer Networks
Igor Shalyminov, Sungjin Lee, Arash Eshghi +1
Goal-oriented dialogue systems are now being widely adopted in industry where it is of key importance to maintain a rapid prototyping cycle for new products and domains. Data-drive…
Hierarchical Multi-Task Natural Language Understanding for Cross-domain Conversational AI: HERMIT NLU
Andrea Vanzo, Emanuele Bastianelli, Oliver Lemon
We present a new neural architecture for wide-coverage Natural Language Understanding in Spoken Dialogue Systems. We develop a hierarchical multi-task architecture, which delivers…
MuMMER: Socially Intelligent Human-Robot Interaction in Public Spaces
Mary Ellen Foster, Bart Craenen, Amol Deshmukh +27
In the EU-funded MuMMER project, we have developed a social robot designed to interact naturally and flexibly with users in public spaces such as a shopping mall. We present the la…
Petri Net Machines for Human-Agent Interaction
Christian Dondrup, Ioannis Papaioannou, Oliver Lemon
Smart speakers and robots become ever more prevalent in our daily lives. These agents are able to execute a wide range of tasks and actions and, therefore, need systems to control…
Few-Shot Dialogue Generation Without Annotated Data: A Transfer Learning Approach
Igor Shalyminov, Sungjin Lee, Arash Eshghi +1
Learning with minimal data is one of the key challenges in the development of practical, production-ready goal-oriented dialogue systems. In a real-world enterprise setting where d…
Neural Response Ranking for Social Conversation: A Data-Efficient Approach
Igor Shalyminov, Ondřej Dušek, Oliver Lemon
The overall objective of 'social' dialogue systems is to support engaging, entertaining, and lengthy conversations on a wide variety of topics, including social chit-chat. Apart fr…
Multi-Task Learning for Domain-General Spoken Disfluency Detection in Dialogue Systems
Igor Shalyminov, Arash Eshghi, Oliver Lemon
Spontaneous spoken dialogue is often disfluent, containing pauses, hesitations, self-corrections and false starts. Processing such phenomena is essential in understanding a speaker…
An Ensemble Model with Ranking for Social Dialogue
Ioannis Papaioannou, Amanda Cercas Curry, Jose L. Part +6
Open-domain social dialogue is one of the long-standing goals of Artificial Intelligence. This year, the Amazon Alexa Prize challenge was announced for the first time, where real c…
The BURCHAK corpus: a Challenge Data Set for Interactive Learning of Visually Grounded Word Meanings
Yanchao Yu, Arash Eshghi, Gregory Mills +1
We motivate and describe a new freely available human-human dialogue dataset for interactive learning of visually grounded word meanings through ostensive definition by a tutor to…
Training an adaptive dialogue policy for interactive learning of visually grounded word meanings
Yanchao Yu, Arash Eshghi, Oliver Lemon
We present a multi-modal dialogue system for interactive learning of perceptually grounded word meanings from a human tutor. The system integrates an incremental, semantic parsing/…
Learning how to learn: an adaptive dialogue agent for incrementally learning visually grounded word meanings
Yanchao Yu, Arash Eshghi, Oliver Lemon
We present an optimised multi-modal dialogue agent for interactive learning of visually grounded word meanings from a human tutor, trained on real human-human tutoring data. Within…
Bootstrapping incremental dialogue systems from minimal data: the generalisation power of dialogue grammars
Arash Eshghi, Igor Shalyminov, Oliver Lemon
We investigate an end-to-end method for automatically inducing task-based dialogue systems from small amounts of unannotated dialogue data. It combines an incremental semantic gram…
Challenging Neural Dialogue Models with Natural Data: Memory Networks Fail on Incremental Phenomena
Igor Shalyminov, Arash Eshghi, Oliver Lemon
Natural, spontaneous dialogue proceeds incrementally on a word-by-word basis; and it contains many sorts of disfluency such as mid-utterance/sentence hesitations, interruptions, an…
Sympathy Begins with a Smile, Intelligence Begins with a Word: Use of Multimodal Features in Spoken Human-Robot Interaction
Jekaterina Novikova, Christian Dondrup, Ioannis Papaioannou +1
Recognition of social signals, from human facial expressions or prosody of speech, is a popular research topic in human-robot interaction studies. There is also a long line of rese…
Bootstrapping incremental dialogue systems: using linguistic knowledge to learn from minimal data
Dimitrios Kalatzis, Arash Eshghi, Oliver Lemon
We present a method for inducing new dialogue systems from very small amounts of unannotated dialogue data, showing how word-level exploration using Reinforcement Learning (RL), co…
Crowd-sourcing NLG Data: Pictures Elicit Better Data
Jekaterina Novikova, Oliver Lemon, Verena Rieser
Recent advances in corpus-based Natural Language Generation (NLG) hold the promise of being easily portable across domains, but require costly training data, consisting of meaning…