1 citations · 3 across the 20 of their papers we have counts for
24 papers
MADS: Multi-Agent Dialog Simulation for Robust Conversational Agents
Arkajyoti Chakraborty, Adam Zernik, Ishika Kulkarni +6
Task-oriented conversational agents remain fragile under realistic conditions, especially when users exhibit non-cooperative behavior. Existing function-calling benchmarks underrep…
ToolRACER: A Robust Agentic Conversation Emulation Resource for Agent Training and Evaluation
Arkajyoti Chakraborty, Aryan Tayal, Ishika Agarwal +5
Task-oriented conversational agents remain fragile under real world conversation scenarios as they rarely follow a predictable script, especially when users exhibit non-cooperative…
From Generative to Discriminative: Lightweight Adaptation of SpeechLLMs for Multimodal Emotion Recognition
Hasindri Watawana, Sergio Burdisso, Esaú Villatoro-Tello +4
SpeechLLMs can recognise emotion by generating a label token, but this approach relies on the LLM's existing output representations. We compare this generative approach with a disc…
Less can be More: What Aspects of Speech Drive End-of-Turn Detection
Rini Sharon, Manickavela A, Kadri Hacioglu +1
In conversational AI, detecting when a speaker has finished talking is crucial for natural turn taking. While recent work incorporates semantics, the relative contribution of diffe…
LLMs Get Smarter from Targeted Synthetic Multilingual Data
Ishika Agarwal, Arkajyoti Charaborty, Tanner Sorensen +2
Language-specific competency (LSC) is the phenomenon of a language model performing better or worse depending on the language of the prompt. In other words, a language model output…
When Synthetic Speech Is All You Have: Better Call GRPO
Shashi Kumar, Yanis Labrak, Hasindri Watawana +5
LLM-based ASR adapted to regulated domains such as banking is bottlenecked by privacy: real speech is costly and legally constrained to collect, making synthetic text-to-speech (TT…
How to Leverage Synthetic Speech for LLM-Based ASR Systems?
Yanis Labrak, Dairazalia Sanchez-Cortes, Sergio Burdisso +9
In regulated domains such as banking and healthcare, where privacy constraints make real speech costly to collect and retain, synthetic speech from modern text-to-speech (TTS) is a…
Closing the Speech-Text Gap with Limited Audio for Effective Domain Adaptation in LLM-Based ASR
Thibault Bañeras-Roux, Sergio Burdisso, Esaú Villatoro-Tello +9
Conventional end-to-end automatic speech recognition (ASR) systems rely on paired speech-text data for domain adaptation. Recent LLM-based ASR architectures connect a speech encode…
Abstract Compression of Conversational Audio Context for LLM-based ASR
Shashi Kumar, Esaú Villatoro-Tello, Sergio Burdisso +7
Standard LLM-based speech recognition systems process utterances in isolation, limiting their ability to leverage conversational context. Here we study whether multimodal context f…
Reducing Prompt Sensitivity in LLM-based Speech Recognition Through Learnable Projection
Sergio Burdisso, Esaú Villatoro-Tello, Shashi Kumar +7
LLM-based automatic speech recognition (ASR), a well-established approach, connects speech foundation models to large language models (LLMs) through a speech-to-LLM projector, yiel…
Text-only adaptation in LLM-based ASR through text denoising
Andrés Carofilis, Sergio Burdisso, Esaú Villatoro-Tello +8
Adapting large language model (LLM)-based automatic speech recognition (ASR) systems to new domains using text-only data is a significant yet underexplored challenge. Standard fine…
Spoken Conversational Agents with Large Language Models
Chao-Han Huck Yang, Andreas Stolcke, Larry Heck
Spoken conversational agents are converging toward voice-native LLMs. This tutorial distills the path from cascaded ASR/NLU to end-to-end, retrieval-and vision-grounded systems. We…
Slot Filling as a Reasoning Task for SpeechLLMs
Kadri Hacioglu, Manjunath K E, Andreas Stolcke
We propose integration of reasoning into speech large language models (speechLLMs) for the end-to-end slot-filling task. Inspired by the recent development of reasoning LLMs, we us…
SpeechLLMs for Large-scale Contextualized Zero-shot Slot Filling
Kadri Hacioglu, Manjunath K E, Andreas Stolcke
Slot filling is a crucial subtask in spoken language understanding (SLU), traditionally implemented as a cascade of speech recognition followed by one or more natural language unde…
TokenVerse++: Towards Flexible Multitask Learning with Dynamic Task Activation
Shashi Kumar, Srikanth Madikeri, Esaú Villatoro-Tello +8
Token-based multitasking frameworks like TokenVerse require all training utterances to have labels for all tasks, hindering their ability to leverage partially annotated datasets a…
Unifying Streaming and Non-streaming Zipformer-based ASR
Bidisha Sharma, Karthik Pandia Durai, Shankar Venkatesan +4
There has been increasing interest in unifying streaming and non-streaming automatic speech recognition (ASR) models to reduce development, training, and deployment costs. We prese…
Better Pseudo-labeling with Multi-ASR Fusion and Error Correction by SpeechLLM
Jeena Prakash, Blessingh Kumar, Kadri Hacioglu +5
Automatic speech recognition (ASR) models rely on high-quality transcribed data for effective training. Generating pseudo-labels for large unlabeled audio datasets often relies on…
Better Semi-supervised Learning for Multi-domain ASR Through Incremental Retraining and Data Filtering
Andres Carofilis, Pradeep Rangappa, Srikanth Madikeri +10
Fine-tuning pretrained ASR models for specific domains is challenging when labeled data is scarce. But unlabeled audio and labeled data from related domains are often available. We…
Efficient Data Selection for Domain Adaptation of ASR Using Pseudo-Labels and Multi-Stage Filtering
Pradeep Rangappa, Andres Carofilis, Jeena Prakash +10
Fine-tuning pretrained ASR models for specific domains is challenging for small organizations with limited labeled data and computational resources. Here, we explore different data…
Improving endpoint detection in end-to-end streaming ASR for conversational speech
Anandh C, Karthik Pandia Durai, Jeena Prakash +6
ASR endpointing (EP) plays a major role in delivering a good user experience in products supporting human or artificial agents in human-human/machine conversations. Transducer-base…
Lightweight Safety Guardrails Using Fine-tuned BERT Embeddings
Aaron Zheng, Mansi Rana, Andreas Stolcke
With the recent proliferation of large language models (LLMs), enterprises have been able to rapidly develop proof-of-concepts and prototypes. As a result, there is a growing need…
Performance evaluation of SLAM-ASR: The Good, the Bad, the Ugly, and the Way Forward
Shashi Kumar, Iuliia Thorbecke, Sergio Burdisso +7
Recent research has demonstrated that training a linear connector between speech foundation encoders and large language models (LLMs) enables this architecture to achieve strong AS…
Provenance: A Light-weight Fact-checker for Retrieval Augmented LLM Generation Output
Hithesh Sankararaman, Mohammed Nasheed Yasin, Tanner Sorensen +2
We present a light-weight approach for detecting nonfactual outputs from retrieval-augmented generation (RAG). Given a context and putative output, we compute a factuality score th…
REFINE on Scarce Data: Retrieval Enhancement through Fine-Tuning via Model Fusion of Embedding Models
Ambuje Gupta, Mrinal Rawat, Andreas Stolcke +1
Retrieval augmented generation (RAG) pipelines are commonly used in tasks such as question-answering (QA), relying on retrieving relevant documents from a vector store computed usi…