43 citations · 169 across the 21 of their papers we have counts for
22 papers
RepFusion: Leveraging Multimodal Priors for Denoising in Representation Space
Xichen Pan, Aashu Singh, Satya Narayan Shukla +3
Large language models (LLMs) are widely used in text-to-image (T2I) systems, but they are typically limited to text encoding, while denoising is handled by newly trained generative…
Unifying Contrastive and Generative Objectives for Visual Understanding and Text-to-Image Generation
Chao Li, Tianhong Li, Sai Vidyaranya Nuthalapati +9
Unifying text-image contrastive learning and text-to-image (T2I) generation in a single end-to-end model is challenging because the two objectives demand opposing masking regimes:…
Xray-Visual Models: Scaling Vision models on Industry Scale Data
Shlok Mishra, Tsung-Yu Lin, Linda Wang +24
We present Xray-Visual, a unified vision model architecture for large-scale image and video understanding trained on industry-scale social media data. Our model leverages over 15 b…
Think Then Embed: Generative Context Improves Multimodal Embedding
Xuanming Cui, Jianpeng Cheng, Hong-you Chen +11
There is a growing interest in Universal Multimodal Embeddings (UME), where models are required to generate task-specific representations. While recent studies show that Multimodal…
StreamMem: Query-Agnostic KV Cache Memory for Streaming Video Understanding
Yanlai Yang, Zhuokai Zhao, Satya Narayan Shukla +4
Multimodal large language models (MLLMs) have made significant progress in visual-language reasoning, but their ability to efficiently handle long videos remains limited. Despite r…
Transfer between Modalities with MetaQueries
Xichen Pan, Satya Narayan Shukla, Aashu Singh +9
Unified multimodal models aim to integrate understanding (text output) and generation (pixel output), but aligning these different modalities within a single architecture often dem…
A Simple and Effective Reinforcement Learning Method for Text-to-Image Diffusion Fine-tuning
Shashank Gupta, Chaitanya Ahuja, Tsung-Yu Lin +4
Reinforcement learning (RL)-based fine-tuning has emerged as a powerful approach for aligning diffusion models with black-box objectives. Proximal policy optimization (PPO) is a po…
BRIDLE: Generalized Self-supervised Learning with Quantization
Hoang M. Nguyen, Satya N. Shukla, Qiang Zhang +6
Self-supervised learning has been a powerful approach for learning meaningful representations from unlabeled data across various domains, reducing the reliance on large labeled dat…
CompCap: Improving Multimodal Large Language Models with Composite Captions
Xiaohui Chen, Satya Narayan Shukla, Mahmoud Azab +8
How well can Multimodal Large Language Models (MLLMs) understand composite images? Composite images (CIs) are synthetic visuals created by merging multiple visual elements, such as…
Learning to Localize Objects Improves Spatial Reasoning in Visual-LLMs
Kanchana Ranasinghe, Satya Narayan Shukla, Omid Poursaeed +2
Integration of Large Language Models (LLMs) into visual domain tasks, resulting in visual-LLMs (V-LLMs), has enabled exceptional performance in vision-language tasks, particularly…
Revisiting Kernel Temporal Segmentation as an Adaptive Tokenizer for Long-form Video Understanding
Mohamed Afham, Satya Narayan Shukla, Omid Poursaeed +3
While most modern video understanding models operate on short-range clips, real-world videos are often several minutes long with semantically consistent segments of variable length…
The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants
Lucas Bandarkar, Davis Liang, Benjamin Muller +7
We present Belebele, a multiple-choice machine reading comprehension (MRC) dataset spanning 122 language variants. Significantly expanding the language coverage of natural language…
Heteroscedastic Temporal Variational Autoencoder For Irregular Time Series
Satya Narayan Shukla, Benjamin M. Marlin
Irregularly sampled time series commonly occur in several domains where they present a significant challenge to standard deep learning models. In this paper, we propose a new deep…
A Survey on Principles, Models and Methods for Learning from Irregularly Sampled Time Series
Satya Narayan Shukla, Benjamin M. Marlin
Irregularly sampled time series data arise naturally in many application domains including biology, ecology, climate science, astronomy, and health. Such data represent fundamental…
Multi-Time Attention Networks for Irregularly Sampled Time Series
Satya Narayan Shukla, Benjamin M. Marlin
Irregular sampling occurs in many time series modeling applications where it presents a significant challenge to standard deep learning models. This work is motivated by the analys…
Gaussian MRF Covariance Modeling for Efficient Black-Box Adversarial Attacks
Anit Kumar Sahu, Satya Narayan Shukla, J. Zico Kolter
We study the problem of generating adversarial examples in a black-box setting, where we only have access to a zeroth order oracle, providing us with loss function evaluations. Alt…
Simple and Efficient Hard Label Black-box Adversarial Attacks in Low Query Budget Regimes
Satya Narayan Shukla, Anit Kumar Sahu, Devin Willmott +1
We focus on the problem of black-box adversarial attacks, where the aim is to generate adversarial examples for deep learning models solely based on information limited to output l…
Integrating Physiological Time Series and Clinical Notes with Deep Learning for Improved ICU Mortality Prediction
Satya Narayan Shukla, Benjamin M. Marlin
Intensive Care Unit Electronic Health Records (ICU EHRs) store multimodal data about patients including clinical notes, sparse and irregularly sampled physiological time series, la…
Assessing the Adversarial Robustness of Monte Carlo and Distillation Methods for Deep Bayesian Neural Network Classification
Meet P. Vadera, Satya Narayan Shukla, Brian Jalaian +1
In this paper, we consider the problem of assessing the adversarial robustness of deep neural network models under both Markov chain Monte Carlo (MCMC) and Bayesian Dark Knowledge…
Black-box Adversarial Attacks with Bayesian Optimization
Satya Narayan Shukla, Anit Kumar Sahu, Devin Willmott +1
We focus on the problem of black-box adversarial attacks, where the aim is to generate adversarial examples using information limited to loss function evaluations of input-output p…
Interpolation-Prediction Networks for Irregularly Sampled Time Series
Satya Narayan Shukla, Benjamin M. Marlin
In this paper, we present a new deep learning architecture for addressing the problem of supervised learning with sparse and irregularly sampled multivariate time series. The archi…
Modeling Irregularly Sampled Clinical Time Series
Satya Narayan Shukla, Benjamin M. Marlin
While the volume of electronic health records (EHR) data continues to grow, it remains rare for hospital systems to capture dense physiological data streams, even in the data-rich…