341 citations · 501 across the 66 of their papers we have counts for
79 papers
Integrating Language Models into Listened and Imagined Speech Decoding from MEG
Maryam Maghsoudi, Sai Samrat Kankanala, Shihab A. Shamma +1
Decoding imagined speech is an important goal for brain-computer interfaces but remains challenging due to weak neural responses, low signal-to-noise ratio, and limited imagined-sp…
JASPER: Joint Audio and Speech Pre-trained Encoder Representations
Geeth George, Ameenudeen P E, Hrishikesh H Pillai +1
Self-supervised learning (SSL) for speech and audio has largely progressed along separate tracks: speech models emphasise time-domain prediction, whereas audio representation learn…
BiG-SURE - Bipartite Graph for Semantic Uncertainty and Reliability Estimation of LLMs
Debarpan Bhattacharya, Malay Phadke, Sriram Ganapathy
Reliable uncertainty estimation is a crucial requirement for deploying large language models (LLMs) and vision-language models (VLMs) in safety-critical settings, especially when t…
ULTRAS -- Unified Learning of Transformer Representations for Audio and Speech Signals
Ameenudeen P E, Charumathi Narayanan, Sriram Ganapathy
Self-supervised learning (SSL) has driven impressive advances in speech processing by adopting time-domain prediction objectives, while audio representation learning frameworks ope…
Benchmarking Speech Systems for Frontline Health Conversations: The DISPLACE-M Challenge
Dhanya E, Ankita Meena, Manas Nanivadekar +11
The DIarization and Speech Processing for LAnguage understanding in Conversational Environments - Medical (DISPLACE-M) challenge introduces a conversational AI benchmark for unders…
A Mixture-of-Experts Model for Multimodal Emotion Recognition in Conversations
Soumya Dutta, Smruthi Balaji, Sriram Ganapathy
Emotion Recognition in Conversations (ERC) presents unique challenges, requiring models to capture the temporal flow of multi-turn dialogues and to effectively integrate cues from…
Benchmarking Humans and Machines on Complex Multilingual Speech Understanding Tasks
Sai Samrat Kankanala, Ram Chandra, Sriram Ganapathy
Auditory attention and selective phase-locking are central to human speech understanding in complex acoustic scenes and cocktail party settings, yet these capabilities in multiling…
FESTA: Functionally Equivalent Sampling for Trust Assessment of Multimodal LLMs
Debarpan Bhattacharya, Apoorva Kulkarni, Sriram Ganapathy
The accurate trust assessment of multimodal large language models (MLLMs) generated predictions, which can enable selective prediction and improve user confidence, is challenging d…
ABHINAYA -- A System for Speech Emotion Recognition In Naturalistic Conditions Challenge
Soumya Dutta, Smruthi Balaji, Varada R +2
Speech emotion recognition (SER) in naturalistic settings remains a challenge due to the intrinsic variability, diverse recording conditions, and class imbalance. As participants i…
Benchmarking and Confidence Evaluation of LALMs For Temporal Reasoning
Debarpan Bhattacharya, Apoorva Kulkarni, Sriram Ganapathy
The popular success of text-based large language models (LLM) has streamlined the attention of the multimodal community to combine other modalities like vision and audio along with…
Spoken Language Understanding on Unseen Tasks With In-Context Learning
Neeraj Agrawal, Sriram Ganapathy
Spoken language understanding (SLU) tasks involve diverse skills that probe the information extraction, classification and/or generation capabilities of models. In this setting, ta…
Textless and Non-Parallel Speech-to-Speech Emotion Style Transfer
Soumya Dutta, Avni Jain, Sriram Ganapathy
Given a pair of source and reference speech recordings, speech-to-speech (S2S) emotion style transfer involves the generation of an output speech that mimics the emotion characteri…
LLM supervised Pre-training for Multimodal Emotion Recognition in Conversations
Soumya Dutta, Sriram Ganapathy
Emotion recognition in conversations (ERC) is challenging due to the multimodal nature of the emotion expression. In this paper, we propose to pretrain a text-based recognition mod…
Uncovering the role of semantic and acoustic cues in normal and dichotic listening
Sai Samrat Kankanala, Akshara Soman, Sriram Ganapathy
Speech comprehension is an involuntary task for the healthy human brain, yet the understanding of the mechanisms underlying this brain functionality remains obscure. In this paper,…
Gradient-free Post-hoc Explainability Using Distillation Aided Learnable Approach
Debarpan Bhattacharya, Amir H. Poorjam, Deepak Mittal +1
The recent advancements in artificial intelligence (AI), with the release of several large models having only query access, make a strong case for explainability of deep models in…
STAB: Speech Tokenizer Assessment Benchmark
Shikhar Vashishth, Harman Singh, Shikhar Bharadwaj +6
Representing speech as discrete tokens provides a framework for transforming speech into a format that closely resembles text, thus enabling the use of speech as an input to the wi…
Leveraging Content and Acoustic Representations for Speech Emotion Recognition
Soumya Dutta, Sriram Ganapathy
Speech emotion recognition (SER), the task of identifying the expression of emotion from spoken content, is challenging due to the difficulty in extracting representations that cap…
Improving Self-supervised Pre-training using Accent-Specific Codebooks
Darshan Prabhu, Abhishek Gupta, Omkar Nitsure +2
Speech accents present a serious challenge to the performance of state-of-the-art end-to-end Automatic Speech Recognition (ASR) systems. Even with self-supervised learning and pre-…
Towards the Next Frontier in Speech Representation Learning Using Disentanglement
Varun Krishna, Sriram Ganapathy
The popular frameworks for self-supervised learning of speech representations have largely focused on frame-level masked prediction of speech regions. While this has shown promisin…
The Second DISPLACE Challenge : DIarization of SPeaker and LAnguage in Conversational Environments
Shareef Babu Kalluri, Prachi Singh, Pratik Roy Chowdhuri +8
The DIarization of SPeaker and LAnguage in Conversational Environments (DISPLACE) 2024 challenge is the second in the series of DISPLACE challenges, which involves tasks of speaker…
Zero Shot Audio to Audio Emotion Transfer With Speaker Disentanglement
Soumya Dutta, Sriram Ganapathy
The problem of audio-to-audio (A2A) style transfer involves replacing the style features of the source audio with those from the target audio while preserving the content related a…
LLM Augmented LLMs: Expanding Capabilities through Composition
Rachit Bansal, Bidisha Samanta, Siddharth Dalmia +6
Foundational models with billions of parameters which have been trained on large corpora of data have demonstrated non-trivial skills in a variety of domains. However, due to their…
End-to-End Supervised Hierarchical Graph Clustering for Speaker Diarization
Prachi Singh, Sriram Ganapathy
Speaker diarization, the task of segmenting an audio recording based on speaker identity, constitutes an important speech pre-processing step for several downstream applications.Th…
Self-Influence Guided Data Reweighting for Language Model Pre-training
Megh Thakkar, Tolga Bolukbasi, Sriram Ganapathy +3
Language Models (LMs) pre-trained with self-supervision on large text corpora have become the default starting point for developing models for various NLP tasks. Once the pre-train…