197 citations · 432 across the 18 of their papers we have counts for
23 papers
DyMU: Dynamic Merging and Virtual Unmerging for Efficient VLMs
Zhenhailong Wang, Senthil Purushwalkam, Caiming Xiong +3
We present DyMU, an efficient, training-free framework that dynamically reduces the computational burden of vision-language models (VLMs) while maintaining high task performance. O…
BLIP3-KALE: Knowledge Augmented Large-Scale Dense Captions
Anas Awadalla, Le Xue, Manli Shu +13
We introduce BLIP3-KALE, a dataset of 218 million image-text pairs that bridges the gap between descriptive synthetic captions and factual web-scale alt-text. KALE augments synthet…
Trust but Verify: Programmatic VLM Evaluation in the Wild
Viraj Prabhu, Senthil Purushwalkam, An Yan +2
Vision-Language Models (VLMs) often generate plausible but incorrect responses to visual queries. However, reliably quantifying the effect of such hallucinations in free-form respo…
FaithEval: Can Your Language Model Stay Faithful to Context, Even If "The Moon is Made of Marshmallows"
Yifei Ming, Senthil Purushwalkam, Shrey Pandit +4
Ensuring faithfulness to context in large language models (LLMs) and retrieval-augmented generation (RAG) systems is crucial for reliable deployment in real-world applications, as…
SFR-RAG: Towards Contextually Faithful LLMs
Xuan-Phi Nguyen, Shrey Pandit, Senthil Purushwalkam +7
Retrieval Augmented Generation (RAG), a paradigm that integrates external contextual information with large language models (LLMs) to enhance factual accuracy and relevance, has em…
xGen-VideoSyn-1: High-fidelity Text-to-Video Synthesis with Compressed Representations
Can Qin, Congying Xia, Krithika Ramakrishnan +16
We present xGen-VideoSyn-1, a text-to-video (T2V) generation model capable of producing realistic scenes from textual descriptions. Building on recent advancements, such as OpenAI'…
xGen-MM (BLIP-3): A Family of Open Large Multimodal Models
Le Xue, Manli Shu, Anas Awadalla +30
This paper introduces BLIP-3, an open framework for developing Large Multimodal Models (LMMs). The framework comprises meticulously curated datasets, a training recipe, model archi…
BootPIG: Bootstrapping Zero-shot Personalized Image Generation Capabilities in Pretrained Diffusion Models
Senthil Purushwalkam, Akash Gokul, Shafiq Joty +1
Recent text-to-image generation models have demonstrated incredible success in generating images that faithfully follow input prompts. However, the requirement of using words to de…
Diffusion Model Alignment Using Direct Preference Optimization
Bram Wallace, Meihua Dang, Rafael Rafailov +7
Large language models (LLMs) are fine-tuned using human comparison data with Reinforcement Learning from Human Feedback (RLHF) methods to make them better aligned with users' prefe…
ConRad: Image Constrained Radiance Fields for 3D Generation from a Single Image
Senthil Purushwalkam, Nikhil Naik
We present a novel method for reconstructing 3D objects from a single RGB image. Our method leverages the latest image generation models to infer the hidden 3D structure while rema…
XGen-7B Technical Report
Erik Nijkamp, Tian Xie, Hiroaki Hayashi +22
Large Language Models (LLMs) have become ubiquitous across various domains, transforming the way we interact with information and conduct research. However, most high-performing LL…
The Challenges of Continuous Self-Supervised Learning
Senthil Purushwalkam, Pedro Morgado, Abhinav Gupta
Self-supervised learning (SSL) aims to eliminate one of the major bottlenecks in representation learning - the need for human annotations. As a result, SSL holds the promise to lea…
The Unsurprising Effectiveness of Pre-Trained Vision Models for Control
Simone Parisi, Aravind Rajeswaran, Senthil Purushwalkam +1
Recent years have seen the emergence of pre-trained representations as a powerful abstraction for AI applications in computer vision, natural language, and speech. However, policy…
The Functional Correspondence Problem
Zihang Lai, Senthil Purushwalkam, Abhinav Gupta
The ability to find correspondences in visual data is the essence of most computer vision tasks. But what are the right correspondences? The task of visual correspondence is well d…
Audio-Visual Floorplan Reconstruction
Senthil Purushwalkam, Sebastian Vicenc Amengual Gari, Vamsi Krishna Ithapu +4
Given only a few glimpses of an environment, how much can we infer about its entire floorplan? Existing methods can map only what is visible or immediately apparent from context, a…
Demystifying Contrastive Self-Supervised Learning: Invariances, Augmentations and Dataset Biases
Senthil Purushwalkam, Abhinav Gupta
Self-supervised representation learning approaches have recently surpassed their supervised learning counterparts on downstream tasks like object detection and image classification…
Aligning Videos in Space and Time
Senthil Purushwalkam, Tian Ye, Saurabh Gupta +1
In this paper, we focus on the task of extracting visual correspondences across videos. Given a query video clip from an action class, we aim to align it with training videos in sp…
Task-Driven Modular Networks for Zero-Shot Compositional Learning
Senthil Purushwalkam, Maximilian Nickel, Abhinav Gupta +1
One of the hallmarks of human intelligence is the ability to compose learned knowledge into novel concepts which can be recognized without a single training example. In contrast, c…
Bounce and Learn: Modeling Scene Dynamics with Real-World Bounces
Senthil Purushwalkam, Abhinav Gupta, Danny M. Kaufman +1
We introduce an approach to model surface properties governing bounces in everyday scenes. Our model learns end-to-end, starting from sensor inputs, to predict post-bounce trajecto…
Pose from Action: Unsupervised Learning of Pose Features based on Motion
Senthil Purushwalkam, Abhinav Gupta
Human actions are comprised of a sequence of poses. This makes videos of humans a rich and dense source of human poses. We propose an unsupervised method to learn pose features fro…
Stochastic Multiple Choice Learning for Training Diverse Deep Ensembles
Stefan Lee, Senthil Purushwalkam, Michael Cogswell +3
Many practical perception systems exist within larger processes that include interactions with users or additional components capable of evaluating the quality of predicted solutio…
Why M Heads are Better than One: Training a Diverse Ensemble of Deep Networks
Stefan Lee, Senthil Purushwalkam, Michael Cogswell +2
Convolutional Neural Networks have achieved state-of-the-art performance on a wide range of tasks. Most benchmarks are led by ensembles of these powerful learners, but ensembling i…
Combining the Best of Graphical Models and ConvNets for Semantic Segmentation
Michael Cogswell, Xiao Lin, Senthil Purushwalkam +1
We present a two-module approach to semantic segmentation that incorporates Convolutional Networks (CNNs) and Graphical Models. Graphical models are used to generate a small (5-30)…