49 citations · 99 across the 30 of their papers we have counts for
32 papers
Text-Prompted CLAP: Learning Text-Conditioned Audio Representations via Contrastive Learning
Mohan Li, Rama Doddipatla, Philip C. Woodland
Contrastive Language-Audio Pretraining (CLAP) aligns text and audio in a shared embedding space, but encoding each modality independently limits its ability to model cross-modal se…
Effectiveness of Chain-of-Thought in Distilling Reasoning Capability from Large Language Models
Cong-Thanh Do, Rama Doddipatla, Kate Knill
Chain-of-Thought (CoT) prompting is a widely used method to improve the reasoning capability of Large Language Models (LLMs). More recently, CoT has been leveraged in Knowledge Dis…
Conditional Multi-Stage Failure Recovery for Embodied Agents
Youmna Farag, Svetlana Stoyanchev, Mohan Li +2
Embodied agents performing complex tasks are susceptible to execution failures, motivating the need for effective failure recovery mechanisms. In this work, we introduce a conditio…
WHISMA: A Speech-LLM to Perform Zero-shot Spoken Language Understanding
Mohan Li, Cong-Thanh Do, Simon Keizer +3
Speech large language models (speech-LLMs) integrate speech and text-based foundation models to provide a unified framework for handling a wide range of downstream tasks. In this p…
Improving Accented Speech Recognition using Data Augmentation based on Unsupervised Text-to-Speech Synthesis
Cong-Thanh Do, Shuhei Imai, Rama Doddipatla +1
This paper investigates the use of unsupervised text-to-speech synthesis (TTS) as a data augmentation method to improve accented speech recognition. TTS systems are trained with a…
Prompting Whisper for QA-driven Zero-shot End-to-end Spoken Language Understanding
Mohan Li, Simon Keizer, Rama Doddipatla
Zero-shot spoken language understanding (SLU) enables systems to comprehend user utterances in new domains without prior exposure to training data. Recent studies often rely on lar…
Semantic Map-based Generation of Navigation Instructions
Chengzu Li, Chao Zhang, Simone Teufel +2
We are interested in the generation of navigation instructions, either in their own right or as training material for robotic navigation task. In this paper, we propose a new appro…
Geodesic interpolation of frame-wise speaker embeddings for the diarization of meeting scenarios
Tobias Cord-Landwehr, Christoph Boeddeker, Cătălin Zorilă +2
We propose a modified teacher-student training for the extraction of frame-wise speaker embeddings that allows for an effective diarization of meeting scenarios containing partiall…
Evaluating Large Language Models for Document-grounded Response Generation in Information-Seeking Dialogues
Norbert Braunschweiler, Rama Doddipatla, Simon Keizer +1
In this paper, we investigate the use of large language models (LLMs) like ChatGPT for document-grounded response generation in the context of information-seeking dialogues. For ev…
Adversarial learning of neural user simulators for dialogue policy optimisation
Simon Keizer, Caroline Dockes, Norbert Braunschweiler +2
Reinforcement learning based dialogue policies are typically trained in interaction with a user simulator. To obtain an effective and robust policy, this simulator should generate…
A Teacher-Student approach for extracting informative speaker embeddings from speech mixtures
Tobias Cord-Landwehr, Christoph Boeddeker, Cătălin Zorilă +2
We introduce a monaural neural speaker embeddings extractor that computes an embedding for each speaker present in a speech mixture. To allow for supervised training, a teacher-stu…
Frame-wise and overlap-robust speaker embeddings for meeting diarization
Tobias Cord-Landwehr, Christoph Boeddeker, Cătălin Zorilă +2
Using a Teacher-Student training approach we developed a speaker embedding extraction system that outputs embeddings at frame rate. Given this high temporal resolution and the fact…
Self-regularised Minimum Latency Training for Streaming Transformer-based Speech Recognition
Mohan Li, Rama Doddipatla, Catalin Zorila
This paper proposes a self-regularised minimum latency training (SR-MLT) method for streaming Transformer-based automatic speech recognition (ASR) systems. In previous works, laten…
Non-autoregressive End-to-end Approaches for Joint Automatic Speech Recognition and Spoken Language Understanding
Mohan Li, Rama Doddipatla
This paper presents the use of non-autoregressive (NAR) approaches for joint automatic speech recognition (ASR) and spoken language understanding (SLU) tasks. The proposed NAR syst…
Multiple-hypothesis RNN-T Loss for Unsupervised Fine-tuning and Self-training of Neural Transducer
Cong-Thanh Do, Mohan Li, Rama Doddipatla
This paper proposes a new approach to perform unsupervised fine-tuning and self-training using unlabeled speech data for recurrent neural network (RNN)-Transducer (RNN-T) end-to-en…
Speaker Reinforcement Using Target Source Extraction for Robust Automatic Speech Recognition
Catalin Zorila, Rama Doddipatla
Improving the accuracy of single-channel automatic speech recognition (ASR) in noisy conditions is challenging. Strong speech enhancement front-ends are available, however, they ty…
On monoaural speech enhancement for automatic recognition of real noisy speech using mixture invariant training
Jisi Zhang, Catalin Zorila, Rama Doddipatla +1
In this paper, we explore an improved framework to train a monoaural neural enhancement model for robust speech recognition. The designed training framework extends the existing mi…
Dialogue Strategy Adaptation to New Action Sets Using Multi-dimensional Modelling
Simon Keizer, Norbert Braunschweiler, Svetlana Stoyanchev +1
A major bottleneck for building statistical spoken dialogue systems for new domains and applications is the need for large amounts of training data. To address this problem, we ado…
Transformer-based Streaming ASR with Cumulative Attention
Mohan Li, Shucong Zhang, Catalin Zorila +1
In this paper, we propose an online attention mechanism, known as cumulative attention (CA), for streaming Transformer-based automatic speech recognition (ASR). Inspired by monoton…
A study on cross-corpus speech emotion recognition and data augmentation
Norbert Braunschweiler, Rama Doddipatla, Simon Keizer +1
Models that can handle a wide range of speakers and acoustic conditions are essential in speech emotion recognition (SER). Often, these models tend to show mixed results when prese…
Monaural source separation: From anechoic to reverberant environments
Tobias Cord-Landwehr, Christoph Boeddeker, Thilo von Neumann +3
Impressive progress in neural network-based single-channel speech source separation has been made in recent years. But those improvements have been mostly reported on anechoic data…
Towards Handling Unconstrained User Preferences in Dialogue
Suraj Pandey, Svetlana Stoyanchev, Rama Doddipatla
A user input to a schema-driven dialogue information navigation system, such as venue search, is typically constrained by the underlying database which restricts the user to specif…
Teacher-Student MixIT for Unsupervised and Semi-supervised Speech Separation
Jisi Zhang, Catalin Zorila, Rama Doddipatla +1
In this paper, we introduce a novel semi-supervised learning framework for end-to-end speech separation. The proposed method first uses mixtures of unseparated sources and the mixt…
Head-synchronous Decoding for Transformer-based Streaming ASR
Mohan Li, Catalin Zorila, Rama Doddipatla
Online Transformer-based automatic speech recognition (ASR) systems have been extensively studied due to the increasing demand for streaming applications. Recently proposed Decoder…