22 citations · 78 across the 18 of their papers we have counts for
19 papers
VideoPoet: A Large Language Model for Zero-Shot Video Generation
Dan Kondratyuk, Lijun Yu, Xiuye Gu +28
We present VideoPoet, a language model capable of synthesizing high-quality video, with matching audio, from a large variety of conditioning signals. VideoPoet employs a decoder-on…
LanSER: Language-Model Supported Speech Emotion Recognition
Taesik Gong, Josh Belanich, Krishna Somandepalli +3
Speech emotion recognition (SER) models typically rely on costly human-labeled data for training, making scaling methods to large speech datasets and nuanced emotion taxonomies dif…
MM-AU:Towards Multimodal Understanding of Advertisement Videos
Digbalay Bose, Rajat Hebbar, Tiantian Feng +3
Advertisement videos (ads) play an integral part in the domain of Internet e-commerce as they amplify the reach of particular products to a broad audience or can serve as a medium…
Contextually-rich human affect perception using multimodal scene information
Digbalay Bose, Rajat Hebbar, Krishna Somandepalli +1
The process of human affect understanding involves the ability to infer person specific emotional states from various sources including images, speech, and language. Affect percept…
Heterogeneous Graph Learning for Acoustic Event Classification
Amir Shirian, Mona Ahmadian, Krishna Somandepalli +1
Heterogeneous graphs provide a compact, efficient, and scalable way to model data involving multiple disparate modalities. This makes modeling audiovisual data using heterogeneous…
A dataset for Audio-Visual Sound Event Detection in Movies
Rajat Hebbar, Digbalay Bose, Krishna Somandepalli +2
Audio event detection is a widely studied audio processing task, with applications ranging from self-driving cars to healthcare. In-the-wild datasets such as Audioset have propelle…
MovieCLIP: Visual Scene Recognition in Movies
Digbalay Bose, Rajat Hebbar, Krishna Somandepalli +5
Longform media such as movies have complex narrative structures, with events spanning a rich variety of ambient visual scenes. Domain specific challenges associated with visual sce…
Visually-aware Acoustic Event Detection using Heterogeneous Graphs
Amir Shirian, Krishna Somandepalli, Victor Sanchez +1
Perception of auditory events is inherently multimodal relying on both audio and visual cues. A large number of existing multimodal approaches process each modality using modality-…
Multitask vocal burst modeling with ResNets and pre-trained paralinguistic Conformers
Josh Belanich, Krishna Somandepalli, Brian Eoff +1
This technical report presents the modeling approaches used in our submission to the ICML Expressive Vocalizations Workshop & Competition multitask track (ExVo-MultiTask). We first…
To train or not to train adversarially: A study of bias mitigation strategies for speaker recognition
Raghuveer Peri, Krishna Somandepalli, Shrikanth Narayanan
Speaker recognition is increasingly used in several everyday applications including smart speakers, customer care centers and other speech-driven analytics. It is crucial to accura…
Self-supervised Graphs for Audio Representation Learning with Limited Labeled Data
Amir Shirian, Krishna Somandepalli, Tanaya Guha
Large scale databases with high-quality manual annotations are scarce in audio domain. We thus explore a self-supervised graph approach to learning audio representations from highl…
Understanding of Emotion Perception from Art
Digbalay Bose, Krishna Somandepalli, Souvik Kundu +3
Computational modeling of the emotions evoked by art in humans is a challenging problem because of the subjective and nuanced nature of art and affective signals. In this paper, we…
Representation of professions in entertainment media: Insights into frequency and sentiment trends through computational text analysis
Sabyasachee Baruah, Krishna Somandepalli, Shrikanth Narayanan
Societal ideas and trends dictate media narratives and cinematic depictions which in turn influences people's beliefs and perceptions of the real world. Media portrayal of culture,…
Robust Character Labeling in Movie Videos: Data Resources and Self-supervised Feature Adaptation
Krishna Somandepalli, Rajat Hebbar, Shrikanth Narayanan
Robust face clustering is a vital step in enabling computational understanding of visual character portrayal in media. Face clustering for long-form content is challenging because…
Generalized Multi-view Shared Subspace Learning using View Bootstrapping
Krishna Somandepalli, Shrikanth Narayanan
A key objective in multi-view learning is to model the information common to multiple parallel views of a class of objects/events to improve downstream learning tasks. In this cont…
Cross modal video representations for weakly supervised active speaker localization
Rahul Sharma, Krishna Somandepalli, Shrikanth Narayanan
An objective understanding of media depictions, such as inclusive portrayals of how much someone is heard and seen on screen such as in film and television, requires the machines t…
An empirical analysis of information encoded in disentangled neural speaker representations
Raghuveer Peri, Haoqi Li, Krishna Somandepalli +2
The primary characteristic of robust speaker representations is that they are invariant to factors of variability not related to speaker identity. Disentanglement of speaker repres…
Robust speaker recognition using unsupervised adversarial invariance
Raghuveer Peri, Monisankha Pal, Arindam Jati +2
In this paper, we address the problem of speaker recognition in challenging acoustic conditions using a novel method to extract robust speaker-discriminative speech representations…
Multimodal Representation Learning using Deep Multiset Canonical Correlation
Krishna Somandepalli, Naveen Kumar, Ruchir Travadi +1
We propose Deep Multiset Canonical Correlation Analysis (dMCCA) as an extension to representation learning using CCA when the underlying signal is observed across multiple (more th…