6 papers
IndicFDB: Benchmarking Full-Duplex Voice Agents across Indian Languages
Rajarshi Roy, Shobhit Banga, Jonathan Raiman +12
Full-duplex voice agents must handle pauses, take turns, backchannel, and respond to user interruptions in real time. Full-Duplex-Bench evaluates these behaviors, but its English-o…
Calibrating LLM Judges for Human and AI Conversations
Maike Züfle, Patrícia Schmidtová, Vilém Zouhar +9
Measuring how successful a conversation is remains difficult, even for humans judging spoken dialogue. We evaluate state-of-the-art LLMs as pointwise and pairwise judges of convers…
Human-1 by Josh Talks: A Full-Duplex Conversational Modeling Framework in Hindi using Real-World Conversations
Bhaskar Singh, Manas Dhir, Shobhit Banga +2
Full-duplex spoken dialogue systems can model natural conversational behaviours such as interruptions, overlaps, and backchannels, yet such systems remain largely unexplored for In…
Preferences of a Voice-First Nation: Large-Scale Pairwise Evaluation and Preference Analysis for TTS in Indian Languages
Srija Anand, Ashwin Sankar, Ishvinder Sethi +10
Crowdsourced pairwise evaluation has emerged as a scalable approach for assessing foundation models. However, applying it to Text to Speech(TTS) introduces high variance due to lin…
Voice of India: A Large-Scale Benchmark for Real-World Speech Recognition in India
Kaushal Bhogale, Manas Dhir, Amritansh Walecha +11
Existing Indic ASR benchmarks often use scripted, clean speech and leaderboard driven evaluation that encourages dataset specific overfitting. In addition, strict single reference…
Benchmarking Speech Systems for Frontline Health Conversations: The DISPLACE-M Challenge
Dhanya E, Ankita Meena, Manas Nanivadekar +11
The DIarization and Speech Processing for LAnguage understanding in Conversational Environments - Medical (DISPLACE-M) challenge introduces a conversational AI benchmark for unders…