5 citations · 20 across the 11 of their papers we have counts for
13 papers
LVLM-Interpret: An Interpretability Tool for Large Vision-Language Models
Gabriela Ben Melech Stan, Estelle Aflalo, Raanan Yehezkel Rohekar +7
In the rapidly evolving landscape of artificial intelligence, multi-modal large language models are emerging as a significant area of interest. These models, which combine various…
LLaVA-Gemma: Accelerating Multimodal Foundation Models with a Compact Language Model
Musashi Hinck, Matthew L. Olson, David Cobbley +2
We train a suite of multimodal foundation models (MMFM) using the popular LLaVA framework with the recently released Gemma family of large language models (LLMs). Of particular int…
ManagerTower: Aggregating the Insights of Uni-Modal Experts for Vision-Language Representation Learning
Xiao Xu, Bei Li, Chenfei Wu +6
Two-Tower Vision-Language (VL) models have shown promising improvements on various downstream VL tasks. Although the most advanced work improves performance by building bridges bet…
MuMUR : Multilingual Multimodal Universal Retrieval
Avinash Madasu, Estelle Aflalo, Gabriela Ben Melech Stan +4
Multi-modal retrieval has seen tremendous progress with the development of vision-language models. However, further improving these models require additional labelled data which is…
VL-InterpreT: An Interactive Visualization Tool for Interpreting Vision-Language Transformers
Estelle Aflalo, Meng Du, Shao-Yen Tseng +4
Breakthroughs in transformer-based models have revolutionized not only the NLP field, but also vision and multimodal systems. However, although visualization and interpretability t…
CALM: Contrastive Aligned Audio-Language Multirate and Multimodal Representations
Vin Sachidananda, Shao-Yen Tseng, Erik Marchi +2
Deriving multimodal representations of audio and lexical inputs is a central problem in Natural Language Understanding (NLU). In this paper, we present Contrastive Aligned Audio-La…
KD-VLP: Improving End-to-End Vision-and-Language Pretraining with Object Knowledge Distillation
Yongfei Liu, Chenfei Wu, Shao-yen Tseng +3
Self-supervised vision-and-language pretraining (VLP) aims to learn transferable multi-modal representations from large-scale image-text data and to achieve strong performances on…
Automatic prediction of suicidal risk in military couples using multimodal interaction cues from couples conversations
Sandeep Nallan Chakravarthula, Md Nasir, Shao-Yen Tseng +6
Suicide is a major societal challenge globally, with a wide range of risk factors, from individual health, psychological and behavioral elements to socio-economic aspects. Military…
Multimodal Embeddings from Language Models
Shao-Yen Tseng, Panayiotis Georgiou, Shrikanth Narayanan
Word embeddings such as ELMo have recently been shown to model word semantics with greater efficacy through contextualized learning on large-scale language corpora, resulting in si…
Behavior Gated Language Models
Prashanth Gurunath Shivakumar, Shao-Yen Tseng, Panayiotis Georgiou +1
Most current language modeling techniques only exploit co-occurrence, semantic and syntactic information from the sequence of words. However, a range of information such as the sta…
Predicting Behavior in Cancer-Afflicted Patient and Spouse Interactions using Speech and Language
Sandeep Nallan Chakravarthula, Haoqi Li, Shao-Yen Tseng +2
Cancer impacts the quality of life of those diagnosed as well as their spouse caregivers, in addition to potentially influencing their day-to-day behaviors. There is evidence that…
Unsupervised Online Multitask Learning of Behavioral Sentence Embeddings
Shao-Yen Tseng, Brian Baucom, Panayiotis Georgiou
Unsupervised learning has been an attractive method for easily deriving meaningful data representations from vast amounts of unlabeled data. These representations, or embeddings, o…
Multiple Instance Deep Learning for Weakly Supervised Small-Footprint Audio Event Detection
Shao-Yen Tseng, Juncheng Li, Yun Wang +3
State-of-the-art audio event detection (AED) systems rely on supervised learning using strongly labeled data. However, this dependence severely limits scalability to large-scale da…