24 citations · 100 across the 31 of their papers we have counts for
34 papers
DialFork: Going Beyond Factuality to Evaluate Procedural Hallucination in Conversational Agents
Sergio Burdisso, Esaú Villatoro-Tello, Dairazalia Sánchez-Cortés +4
Large language models (LLMs) deployed as conversational agents are increasingly expected to follow procedural logic. While factual hallucination has received extensive attention, a…
Rethinking Human-Aligned Evaluation: An Analysis of Semantic Metrics Beyond WER
Hritika Sharma, Thibault Bañeras-Roux, Alessandra Pinto +4
Word Error Rate (WER), the most commonly used metric for Automatic Speech Recognition (ASR), treats every lexical deviation from the reference as equally costly, regardless of whet…
From Generative to Discriminative: Lightweight Adaptation of SpeechLLMs for Multimodal Emotion Recognition
Hasindri Watawana, Sergio Burdisso, Esaú Villatoro-Tello +4
SpeechLLMs can recognise emotion by generating a label token, but this approach relies on the LLM's existing output representations. We compare this generative approach with a disc…
When Synthetic Speech Is All You Have: Better Call GRPO
Shashi Kumar, Yanis Labrak, Hasindri Watawana +5
LLM-based ASR adapted to regulated domains such as banking is bottlenecked by privacy: real speech is costly and legally constrained to collect, making synthetic text-to-speech (TT…
How to Leverage Synthetic Speech for LLM-Based ASR Systems?
Yanis Labrak, Dairazalia Sanchez-Cortes, Sergio Burdisso +9
In regulated domains such as banking and healthcare, where privacy constraints make real speech costly to collect and retain, synthetic speech from modern text-to-speech (TTS) is a…
Closing the Speech-Text Gap with Limited Audio for Effective Domain Adaptation in LLM-Based ASR
Thibault Bañeras-Roux, Sergio Burdisso, Esaú Villatoro-Tello +9
Conventional end-to-end automatic speech recognition (ASR) systems rely on paired speech-text data for domain adaptation. Recent LLM-based ASR architectures connect a speech encode…
Abstract Compression of Conversational Audio Context for LLM-based ASR
Shashi Kumar, Esaú Villatoro-Tello, Sergio Burdisso +7
Standard LLM-based speech recognition systems process utterances in isolation, limiting their ability to leverage conversational context. Here we study whether multimodal context f…
When Consistency Becomes Bias: Interviewer Effects in Semi-Structured Clinical Interviews
Hasindri Watawana, Sergio Burdisso, Diego A. Moreno-Galván +4
Automatic depression detection from doctor-patient conversations has gained momentum thanks to the availability of public corpora and advances in language modeling. However, interp…
Reducing Prompt Sensitivity in LLM-based Speech Recognition Through Learnable Projection
Sergio Burdisso, Esaú Villatoro-Tello, Shashi Kumar +7
LLM-based automatic speech recognition (ASR), a well-established approach, connects speech foundation models to large language models (LLMs) through a speech-to-LLM projector, yiel…
Text-only adaptation in LLM-based ASR through text denoising
Andrés Carofilis, Sergio Burdisso, Esaú Villatoro-Tello +8
Adapting large language model (LLM)-based automatic speech recognition (ASR) systems to new domains using text-only data is a significant yet underexplored challenge. Standard fine…
SDialog: A Python Toolkit for End-to-End Agent Building, User Simulation, Dialog Generation, and Evaluation
Sergio Burdisso, Séverin Baroudi, Yanis Labrak +8
We present SDialog, an MIT-licensed open-source Python toolkit that unifies dialog generation, evaluation and mechanistic interpretability into a single end-to-end framework for bu…
TokenVerse++: Towards Flexible Multitask Learning with Dynamic Task Activation
Shashi Kumar, Srikanth Madikeri, Esaú Villatoro-Tello +8
Token-based multitasking frameworks like TokenVerse require all training utterances to have labels for all tasks, hindering their ability to leverage partially annotated datasets a…
Better Semi-supervised Learning for Multi-domain ASR Through Incremental Retraining and Data Filtering
Andres Carofilis, Pradeep Rangappa, Srikanth Madikeri +10
Fine-tuning pretrained ASR models for specific domains is challenging when labeled data is scarce. But unlabeled audio and labeled data from related domains are often available. We…
Efficient Data Selection for Domain Adaptation of ASR Using Pseudo-Labels and Multi-Stage Filtering
Pradeep Rangappa, Andres Carofilis, Jeena Prakash +10
Fine-tuning pretrained ASR models for specific domains is challenging for small organizations with limited labeled data and computational resources. Here, we explore different data…
SDialog: A Python Toolkit for End-to-End Agent Building, User Simulation, Dialog Generation, and Evaluation
Sergio Burdisso, Séverin Baroudi, Yanis Labrak +9
We present SDialog, an MIT-licensed open-source Python toolkit that unifies dialog generation, evaluation and mechanistic interpretability into a single end-to-end framework for bu…
Temporal fine-tuning for early risk detection
Horacio Thompson, Esaú Villatoro-Tello, Manuel Montes-y-Gómez +1
Early Risk Detection (ERD) on the Web aims to identify promptly users facing social and health issues. Users are analyzed post-by-post, and it is necessary to guarantee correct and…
Performance evaluation of SLAM-ASR: The Good, the Bad, the Ugly, and the Way Forward
Shashi Kumar, Iuliia Thorbecke, Sergio Burdisso +7
Recent research has demonstrated that training a linear connector between speech foundation encoders and large language models (LLMs) enables this architecture to achieve strong AS…
Mapping the Media Landscape: Predicting Factual Reporting and Political Bias Through Web Interactions
Dairazalia Sánchez-Cortés, Sergio Burdisso, Esaú Villatoro-Tello +1
Bias assessment of news sources is paramount for professionals, organizations, and researchers who rely on truthful evidence for information gathering and reporting. While certain…
Fast Streaming Transducer ASR Prototyping via Knowledge Distillation with Whisper
Iuliia Thorbecke, Juan Zuluaga-Gomez, Esaú Villatoro-Tello +6
The training of automatic speech recognition (ASR) with little to no supervised data remains an open question. In this work, we demonstrate that streaming Transformer-Transducer (T…
Unifying Global and Near-Context Biasing in a Single Trie Pass
Iuliia Thorbecke, Esaú Villatoro-Tello, Juan Zuluaga-Gomez +9
Despite the success of end-to-end automatic speech recognition (ASR) models, challenges persist in recognizing rare, out-of-vocabulary words - including named entities (NE) - and i…
TokenVerse: Towards Unifying Speech and NLP Tasks via Transducer-based ASR
Shashi Kumar, Srikanth Madikeri, Juan Zuluaga-Gomez +6
In traditional conversational intelligence from speech, a cascaded pipeline is used, involving tasks such as voice activity detection, diarization, transcription, and subsequent pr…
XLSR-Transducer: Streaming ASR for Self-Supervised Pretrained Models
Shashi Kumar, Srikanth Madikeri, Juan Zuluaga-Gomez +5
Self-supervised pretrained models exhibit competitive performance in automatic speech recognition on finetuning, even with limited in-domain supervised data. However, popular pretr…
DAIC-WOZ: On the Validity of Using the Therapist's prompts in Automatic Depression Detection from Clinical Interviews
Sergio Burdisso, Ernesto Reyes-Ramírez, Esaú Villatoro-Tello +3
Automatic depression detection from conversational data has gained significant interest in recent years. The DAIC-WOZ dataset, interviews conducted by a human-controlled virtual ag…
Reliability Estimation of News Media Sources: Birds of a Feather Flock Together
Sergio Burdisso, Dairazalia Sánchez-Cortés, Esaú Villatoro-Tello +1
Evaluating the reliability of news sources is a routine task for journalists and organizations committed to acquiring and disseminating accurate information. Recent research has sh…