21 papers
Indic DiarBench: A Multilingual Joint Diarization and ASR Benchmark for Indian Languages
Deovrat Mehendale, Aditya Mehndiratta, Dhruv Rathi +2
In this work, we introduce Indic DiarBench, a speaker diarization and ASR benchmark dataset spanning all 22 scheduled languages of India. This corpus comprises approximately 108 ho…
Voice of India: A Large-Scale Benchmark for Real-World Speech Recognition in India
Kaushal Bhogale, Manas Dhir, Amritansh Walecha +11
Existing Indic ASR benchmarks often use scripted, clean speech and leaderboard driven evaluation that encourages dataset specific overfitting. In addition, strict single reference…
IndicContextEval: A Benchmark for Evaluating Context Utilisation in Audio Large Language Models Across 8 Indic Languages
Sakshi Joshi, Dhruv Subhash Rathi, Sanskar Singh +4
AudioLLMs enable speech recognition conditioned on textual prompts such as domain descriptions or entity lists. However, it remains unclear whether these models genuinely utilise s…
Preferences of a Voice-First Nation: Large-Scale Pairwise Evaluation and Preference Analysis for TTS in Indian Languages
Srija Anand, Ashwin Sankar, Ishvinder Sethi +10
Crowdsourced pairwise evaluation has emerged as a scalable approach for assessing foundation models. However, applying it to Text to Speech(TTS) introduces high variance due to lin…
Seeing Isn't Believing: Uncovering Blind Spots in Evaluator Vision-Language Models
Mohammed Safi Ur Rahman Khan, Sanjay Suryanarayanan, Tushar Anand +1
Large Vision-Language Models (VLMs) are increasingly used to evaluate outputs of other models, for image-to-text (I2T) tasks such as visual question answering, and text-to-image (T…
Towards Orthographically-Informed Evaluation of Speech Recognition Systems for Indian Languages
Kaushal Santosh Bhogale, Tahir Javed, Greeshma Susan John +4
Evaluating ASR systems for Indian languages is challenging due to spelling variations, suffix splitting flexibility, and non-standard spellings in code-mixed words. Traditional Wor…