5 citations · 6 across the 11 of their papers we have counts for
14 papers
RePlay: Retrieval-Based Voice Playback for Multi-Turn spoken dialogue
Sathvik Udupa, Naveen Kumar, Ryan Folmsbee
Many voice interaction applications require exact control over both the content and delivery of responses, typically using pre-recorded lines. Recent full-duplex models respond wit…
TurnBench: A Multi-Domain Benchmark for Turn-Taking Dynamics in Spoken Dialogue
Freeman Jiang, Ramon Sanabria, Soham Deshmukh +16
Speakers in natural conversation take turns speaking and listening, deciding in real time when to take, hold, or yield the floor. However, turn-taking evaluation remains limited du…
Grounding Spoken LLMs in Multi-Speaker Audio via Diarization Conditioning
Alexander Polok, Samuele Cornell, Sathvik Udupa +3
We propose diarization-conditioned spoken language models (SLMs), a strategy for extending SLMs to far-field multi-talker audio. Rather than adapting the decoder via Serialized Out…
Endpoint Anticipation for Low-Latency Spoken Dialogue
Sathvik Udupa, Shinji Watanabe, Petr Schwarz +1
While low-latency interaction is critical for spoken dialogue, cascaded architectures are often bottlenecked by reactive turn-completion detection. We propose Endpoint Anticipation…
VAANI: Capturing the language landscape for an inclusive digital India
Sujith Pulikodan, Abhayjeet Singh, Agneedh Basu +17
Voice based technologies have the potential to bridge digital accessibility gaps; however, existing datasets fail to capture the linguistic and regional diversity of Indic language…
ORCA: Open-ended Response Correctness Assessment for Audio Question Answering
Šimon Sedláček, Sara Barahona, Bolaji Yusuf +9
Reliable assessment of the abilities of large audio language models (LALMs) is essential to advancing the state of the art. As benchmarks rapidly evolve to incorporate complex reas…
MMAU-Pro: A Challenging and Comprehensive Benchmark for Holistic Evaluation of Audio General Intelligence
Sonal Kumar, Šimon Sedláček, Vaibhavi Lokegaonkar +31
Audio comprehension-including speech, non-speech sounds, and music-is essential for achieving human-level intelligence. Consequently, AI agents must demonstrate holistic audio unde…
Streaming Endpointer for Spoken Dialogue using Neural Audio Codecs and Label-Delayed Training
Sathvik Udupa, Shinji Watanabe, Petr Schwarz +1
Accurate, low-latency endpointing is crucial for effective spoken dialogue systems. While traditional endpointers often rely on spectrum-based audio features, this work proposes re…
Discovering phoneme-specific critical articulators through a data-driven approach
Jesuraj Bandekar, Sathvik Udupa, Prasanta Kumar Ghosh
We propose an approach for learning critical articulators for phonemes through a machine learning approach. We formulate the learning with three models trained end to end. First, w…
Speaking rate attention-based duration prediction for speed control TTS
Jesuraj Bandekar, Sathvik Udupa, Abhayjeet Singh +6
With the advent of high-quality speech synthesis, there is a lot of interest in controlling various prosodic attributes of speech. Speaking rate is an essential attribute towards m…
Model Adaptation for ASR in low-resource Indian Languages
Abhayjeet Singh, Arjun Singh Mehta, Ashish Khuraishi K S +17
Automatic speech recognition (ASR) performance has improved drastically in recent years, mainly enabled by self-supervised learning (SSL) based acoustic models such as wav2vec2 and…
Real-Time MRI Video synthesis from time aligned phonemes with sequence-to-sequence networks
Sathvik Udupa, Prasanta Kumar Ghosh
Real-Time Magnetic resonance imaging (rtMRI) of the midsagittal plane of the mouth is of interest for speech production research. In this work, we focus on estimating utterance lev…
Improved acoustic-to-articulatory inversion using representations from pretrained self-supervised learning models
Sathvik Udupa, Siddarth C, Prasanta Kumar Ghosh
In this work, we investigate the effectiveness of pretrained Self-Supervised Learning (SSL) features for learning the mapping for acoustic to articulatory inversion (AAI). Signal p…
Estimating articulatory movements in speech production with transformer networks
Sathvik Udupa, Anwesha Roy, Abhayjeet Singh +2
We estimate articulatory movements in speech production from different modalities - acoustics and phonemes. Acoustic-to articulatory inversion (AAI) is a sequence-to-sequence task.…