326 citations · 717 across the 90 of their papers we have counts for
8 papers · 1 filter
Vimarsha: Faithful ASR Evaluation for Indian Languages with Demographic Diversity, In-the-Wild Audio and Spelling Variations
Kaushal Santosh Bhogale, Srija Anand, Sadakopa Ramakrishnan Thothathiri +3
Evaluation benchmarks for Indian language automatic speech recognition (ASR) suffer from two systematic biases: optimistic scores from clean, controlled audio conditions, and pessi…
IndicQE-APE: A Consolidated Benchmark for Quality Estimation and Automatic Post-Editing for Indic Languages
Diptesh Kanojia, Archchana Sindhujan, Sourabh Deoghare +14
Indic quality estimation (QE) and automatic post-editing (APE) data is spread across separate releases, so no single resource supports training and evaluation across tasks and lang…
Indic DiarBench: A Multilingual Joint Diarization and ASR Benchmark for Indian Languages
Deovrat Mehendale, Aditya Mehndiratta, Dhruv Rathi +2
In this work, we introduce Indic DiarBench, a speaker diarization and ASR benchmark dataset spanning all 22 scheduled languages of India. This corpus comprises approximately 108 ho…
IndicContextEval: A Benchmark for Evaluating Context Utilisation in Audio Large Language Models Across 8 Indic Languages
Sakshi Joshi, Dhruv Subhash Rathi, Sanskar Singh +4
AudioLLMs enable speech recognition conditioned on textual prompts such as domain descriptions or entity lists. However, it remains unclear whether these models genuinely utilise s…
Seeing Isn't Believing: Uncovering Blind Spots in Evaluator Vision-Language Models
Mohammed Safi Ur Rahman Khan, Sanjay Suryanarayanan, Tushar Anand +1
Large Vision-Language Models (VLMs) are increasingly used to evaluate outputs of other models, for image-to-text (I2T) tasks such as visual question answering, and text-to-image (T…
Preferences of a Voice-First Nation: Large-Scale Pairwise Evaluation and Preference Analysis for TTS in Indian Languages
Srija Anand, Ashwin Sankar, Ishvinder Sethi +10
Crowdsourced pairwise evaluation has emerged as a scalable approach for assessing foundation models. However, applying it to Text to Speech(TTS) introduces high variance due to lin…
Voice of India: A Large-Scale Benchmark for Real-World Speech Recognition in India
Kaushal Bhogale, Manas Dhir, Amritansh Walecha +11
Existing Indic ASR benchmarks often use scripted, clean speech and leaderboard driven evaluation that encourages dataset specific overfitting. In addition, strict single reference…
Towards Orthographically-Informed Evaluation of Speech Recognition Systems for Indian Languages
Kaushal Santosh Bhogale, Tahir Javed, Greeshma Susan John +4
Evaluating ASR systems for Indian languages is challenging due to spelling variations, suffix splitting flexibility, and non-standard spellings in code-mixed words. Traditional Wor…