66 citations · 155 across the 27 of their papers we have counts for
31 papers
SayCoNav: Utilizing Large Language Models for Adaptive Collaboration in Decentralized Multi-Robot Navigation
Abhinav Rajvanshi, Pritish Sahu, Tixiao Shan +2
Adaptive collaboration is critical to a team of autonomous robots to perform complicated navigation tasks in large-scale unknown environments. An effective collaboration strategy s…
Pelican: Correcting Hallucination in Vision-LLMs via Claim Decomposition and Program of Thought Verification
Pritish Sahu, Karan Sikka, Ajay Divakaran
Large Visual Language Models (LVLMs) struggle with hallucinations in visual instruction following task(s), limiting their trustworthiness and real-world applicability. We propose P…
A Video is Worth 10,000 Words: Training and Benchmarking with Diverse Captions for Better Long Video Retrieval
Matthew Gwilliam, Michael Cogswell, Meng Ye +3
Existing long video retrieval systems are trained and tested in the paragraph-to-video retrieval regime, where every long video is described by a single long paragraph. This neglec…
DRESS: Instructing Large Vision-Language Models to Align and Interact with Humans via Natural Language Feedback
Yangyi Chen, Karan Sikka, Michael Cogswell +2
We present DRESS, a large vision language model (LVLM) that innovatively exploits Natural Language feedback (NLF) from Large Language Models to enhance its alignment and interactio…
Demonstrations Are All You Need: Advancing Offensive Content Paraphrasing using In-Context Learning
Anirudh Som, Karan Sikka, Helen Gent +3
Paraphrasing of offensive content is a better alternative to content removal and helps improve civility in a communication environment. Supervised paraphrasers; however, rely heavi…
Measuring and Improving Chain-of-Thought Reasoning in Vision-Language Models
Yangyi Chen, Karan Sikka, Michael Cogswell +2
Vision-language models (VLMs) have recently demonstrated strong efficacy as visual assistants that can parse natural queries about the visual content and generate human-like output…
SayNav: Grounding Large Language Models for Dynamic Planning to Navigation in New Environments
Abhinav Rajvanshi, Karan Sikka, Xiao Lin +3
Semantic reasoning and dynamic planning capabilities are crucial for an autonomous agent to perform complex navigation tasks in unknown environments. It requires a large amount of…
TIJO: Trigger Inversion with Joint Optimization for Defending Multimodal Backdoored Models
Indranil Sur, Karan Sikka, Matthew Walmer +5
We present a Multimodal Backdoor Defense technique TIJO (Trigger Inversion using Joint Optimization). Recent work arXiv:2112.07668 has demonstrated successful backdoor attacks on m…
Predicting Information Pathways Across Online Communities
Yiqiao Jin, Yeon-Chang Lee, Kartik Sharma +4
The problem of community-level information pathway prediction (CLIPP) aims at predicting the transmission trajectory of content across online communities. A successful solution to…
Multilingual Content Moderation: A Case Study on Reddit
Meng Ye, Karan Sikka, Katherine Atwell +3
Content moderation is the process of flagging content based on pre-defined platform rules. There has been a growing need for AI moderators to safeguard users as well as protect the…
Dual-Key Multimodal Backdoors for Visual Question Answering
Matthew Walmer, Karan Sikka, Indranil Sur +2
The success of deep learning has enabled advances in multimodal tasks that require non-trivial fusion of multiple input domains. Although multimodal models have shown potential in…
Challenges in Procedural Multimodal Machine Comprehension:A Novel Way To Benchmark
Pritish Sahu, Karan Sikka, Ajay Divakaran
We focus on Multimodal Machine Reading Comprehension (M3C) where a model is expected to answer questions based on given passage (or context), and the context and the questions can…
Towards Solving Multimodal Comprehension
Pritish Sahu, Karan Sikka, Ajay Divakaran
This paper targets the problem of procedural multimodal machine comprehension (M3C). This task requires an AI to comprehend given steps of multimodal instructions and then answer q…
Detecting Trojaned DNNs Using Counterfactual Attributions
Karan Sikka, Indranil Sur, Susmit Jha +2
We target the problem of detecting Trojans or backdoors in DNNs. Such models behave normally with typical inputs but produce specific incorrect predictions for inputs poisoned with…
Zero-Shot Learning with Knowledge Enhanced Visual Semantic Embeddings
Karan Sikka, Jihua Huang, Andrew Silberfarb +6
We improve zero-shot learning (ZSL) by incorporating common-sense knowledge in DNNs. We propose Common-Sense based Neuro-Symbolic Loss (CSNL) that formulates prior knowledge as nov…
RGB2LIDAR: Towards Solving Large-Scale Cross-Modal Visual Localization
Niluthpol Chowdhury Mithun, Karan Sikka, Han-Pang Chiu +2
We study an important, yet largely unexplored problem of large-scale cross-modal visual localization by matching ground RGB images to a geo-referenced aerial LIDAR 3D point cloud (…
Deep Adaptive Semantic Logic (DASL): Compiling Declarative Knowledge into Deep Neural Networks
Karan Sikka, Andrew Silberfarb, John Byrnes +4
We introduce Deep Adaptive Semantic Logic (DASL), a novel framework for automating the generation of deep neural networks that incorporates user-provided formal knowledge to improv…
Sunny and Dark Outside?! Improving Answer Consistency in VQA through Entailed Question Generation
Arijit Ray, Karan Sikka, Ajay Divakaran +2
While models for Visual Question Answering (VQA) have steadily improved over the years, interacting with one quickly reveals that these models lack consistency. For instance, if a…
FoodX-251: A Dataset for Fine-grained Food Classification
Parneet Kaur, Karan Sikka, Weijun Wang +2
Food classification is a challenging problem due to the large number of categories, high visual similarity between different foods, as well as the lack of datasets for training sta…
Deep Unified Multimodal Embeddings for Understanding both Content and Users in Social Media Networks
Karan Sikka, Lucas Van Bramer, Ajay Divakaran
There has been an explosion of multimodal content generated on social media networks in the last few years, which has necessitated a deeper understanding of social media content an…
Integrating Text and Image: Determining Multimodal Document Intent in Instagram Posts
Julia Kruk, Jonah Lubin, Karan Sikka +3
Computing author intent from multimodal data like Instagram posts requires modeling a complex relationship between text and image. For example, a caption might evoke an ironic cont…
Align2Ground: Weakly Supervised Phrase Grounding Guided by Image-Caption Alignment
Samyak Datta, Karan Sikka, Anirban Roy +3
We address the problem of grounding free-form textual phrases by using weak supervision from image-caption pairs. We propose a novel end-to-end model that uses caption-to-image ret…
Semantically-Aware Attentive Neural Embeddings for Image-based Visual Localization
Zachary Seymour, Karan Sikka, Han-Pang Chiu +2
We present an approach that combines appearance and semantic information for 2D image-based localization (2D-VL) across large perceptual changes and time lags. Compared to appearan…
Understanding Visual Ads by Aligning Symbols and Objects using Co-Attention
Karuna Ahuja, Karan Sikka, Anirban Roy +1
We tackle the problem of understanding visual ads where given an ad image, our goal is to rank appropriate human generated statements describing the purpose of the ad. This problem…