43 citations · 236 across the 34 of their papers we have counts for
34 papers
You Talkin to Me?: A Network Analysis of Gendered Speaker-Addressee Patterns in Film Screenplays
Samin Khan, Camilla Griffiths, Shrikanth Narayanan +2
Objective: This paper investigates the gendered structure of speaker addressee relationships in film dialogue, asking not merely who speaks, but who is spoken to and how conversati…
Robust Self Supervised Speech Embeddings for Child-Adult Classification in Interactions involving Children with Autism
Rimita Lahiri, Tiantian Feng, Rajat Hebbar +3
We address the problem of detecting who spoke when in child-inclusive spoken interactions i.e., automatic child-adult speaker classification. Interactions involving children are ri…
FedMultimodal: A Benchmark For Multimodal Federated Learning
Tiantian Feng, Digbalay Bose, Tuo Zhang +6
Over the past few years, Federated Learning (FL) has become an emerging machine learning technique to tackle data privacy challenges through collaborative training. In the Federate…
Unlocking Foundation Models for Privacy-Enhancing Speech Understanding: An Early Study on Low Resource Speech Training Leveraging Label-guided Synthetic Speech Content
Tiantian Feng, Digbalay Bose, Xuan Shi +1
Automatic Speech Understanding (ASU) leverages the power of deep learning models for accurate interpretation of human speech, leading to a wide range of speech applications that en…
PEFT-SER: On the Use of Parameter Efficient Transfer Learning Approaches For Speech Emotion Recognition Using Pre-trained Speech Models
Tiantian Feng, Shrikanth Narayanan
Many recent studies have focused on fine-tuning pre-trained models for speech emotion recognition (SER), resulting in promising performance compared to traditional methods that rel…
GPT-FL: Generative Pre-trained Model-Assisted Federated Learning
Tuo Zhang, Tiantian Feng, Samiul Alam +5
In this work, we propose GPT-FL, a generative pre-trained model-assisted federated learning (FL) framework. At its core, GPT-FL leverages generative pre-trained models to generate…
Understanding Spoken Language Development of Children with ASD Using Pre-trained Speech Embeddings
Anfeng Xu, Rajat Hebbar, Rimita Lahiri +5
Speech processing techniques are useful for analyzing speech and language development in children with Autism Spectrum Disorder (ASD), who are often varied and delayed in acquiring…
TrustSER: On the Trustworthiness of Fine-tuning Pre-trained Speech Embeddings For Speech Emotion Recognition
Tiantian Feng, Rajat Hebbar, Shrikanth Narayanan
Recent studies have explored the use of pre-trained embeddings for speech emotion recognition (SER), achieving comparable performance to conventional methods that rely on low-level…
Designing and Evaluating Speech Emotion Recognition Systems: A reality check case study with IEMOCAP
Nikolaos Antoniou, Athanasios Katsamanis, Theodoros Giannakopoulos +1
There is an imminent need for guidelines and standard test sets to allow direct and fair comparisons of speech emotion recognition (SER). While resources, such as the Interactive E…
Contextually-rich human affect perception using multimodal scene information
Digbalay Bose, Rajat Hebbar, Krishna Somandepalli +1
The process of human affect understanding involves the ability to infer person specific emotional states from various sources including images, speech, and language. Affect percept…
A dataset for Audio-Visual Sound Event Detection in Movies
Rajat Hebbar, Digbalay Bose, Krishna Somandepalli +2
Audio event detection is a widely studied audio processing task, with applications ranging from self-driving cars to healthcare. In-the-wild datasets such as Audioset have propelle…
A Review of Speech-centric Trustworthy Machine Learning: Privacy, Safety, and Fairness
Tiantian Feng, Rajat Hebbar, Nicholas Mehlman +3
Speech-centric machine learning systems have revolutionized many leading domains ranging from transportation and healthcare to education and defense, profoundly changing how people…
A Context-Aware Computational Approach for Measuring Vocal Entrainment in Dyadic Conversations
Rimita Lahiri, Md Nasir, Catherine Lord +2
Vocal entrainment is a social adaptation mechanism in human interaction, knowledge of which can offer useful insights to an individual's cognitive-behavioral characteristics. We pr…
Can Knowledge of End-to-End Text-to-Speech Models Improve Neural MIDI-to-Audio Synthesis Systems?
Xuan Shi, Erica Cooper, Xin Wang +2
With the similarity between music and speech synthesis from symbolic input and the rapid development of text-to-speech (TTS) techniques, it is worthwhile to explore ways to improve…
Using Emotion Embeddings to Transfer Knowledge Between Emotions, Languages, and Annotation Formats
Georgios Chochlakis, Gireesh Mahajan, Sabyasachee Baruah +3
The need for emotional inference from text continues to diversify as more and more disciplines integrate emotions into their theories and applications. These needs include inferrin…
On the Role of Visual Context in Enriching Music Representations
Kleanthis Avramidis, Shanti Stewart, Shrikanth Narayanan
Human perception and experience of music is highly context-dependent. Contextual variability contributes to differences in how we interpret and interact with music, challenging the…
Leveraging Open Data and Task Augmentation to Automated Behavioral Coding of Psychotherapy Conversations in Low-Resource Scenarios
Zhuohao Chen, Nikolaos Flemotomos, Zac E. Imel +2
In psychotherapy interactions, the quality of a session is assessed by codifying the communicative behaviors of participants during the conversation through manual observation and…
MovieCLIP: Visual Scene Recognition in Movies
Digbalay Bose, Rajat Hebbar, Krishna Somandepalli +5
Longform media such as movies have complex narrative structures, with events spanning a rich variety of ambient visual scenes. Domain specific challenges associated with visual sce…
Leveraging Label Correlations in a Multi-label Setting: A Case Study in Emotion
Georgios Chochlakis, Gireesh Mahajan, Sabyasachee Baruah +3
Detecting emotions expressed in text has become critical to a range of fields. In this work, we investigate ways to exploit label correlations in multi-label emotion recognition mo…
FedAudio: A Federated Learning Benchmark for Audio Tasks
Tuo Zhang, Tiantian Feng, Samiul Alam +4
Federated learning (FL) has gained substantial attention in recent years due to the data privacy concerns related to the pervasiveness of consumer devices that continuously collect…
The 2022 Far-field Speaker Verification Challenge: Exploring domain mismatch and semi-supervised learning under the far-field scenario
Xiaoyi Qin, Ming Li, Hui Bu +2
FFSVC2022 is the second challenge of far-field speaker verification. FFSVC2022 provides the fully-supervised far-field speaker verification to further explore the far-field scenari…
VAuLT: Augmenting the Vision-and-Language Transformer for Sentiment Classification on Social Media
Georgios Chochlakis, Tejas Srinivasan, Jesse Thomason +1
We propose the Vision-and-Augmented-Language Transformer (VAuLT). VAuLT is an extension of the popular Vision-and-Language Transformer (ViLT), and improves performance on vision-an…
User-Level Differential Privacy against Attribute Inference Attack of Speech Emotion Recognition in Federated Learning
Tiantian Feng, Raghuveer Peri, Shrikanth Narayanan
Many existing privacy-enhanced speech emotion recognition (SER) frameworks focus on perturbing the original speech data through adversarial training within a centralized machine le…
Mel Frequency Spectral Domain Defenses against Adversarial Attacks on Speech Recognition Systems
Nicholas Mehlman, Anirudh Sreeram, Raghuveer Peri +1
A variety of recent works have looked into defenses for deep neural networks against adversarial attacks particularly within the image processing domain. Speech processing applicat…