66 citations · 168 across the 35 of their papers we have counts for
18 papers · 1 filter
Class Prototypes based Contrastive Learning for Classifying Multi-Label and Fine-Grained Educational Videos
Rohit Gupta, Anirban Roy, Claire Christensen +6
The recent growth in the consumption of online media by children during early childhood necessitates data-driven tools enabling educators to filter out appropriate educational cont…
TIJO: Trigger Inversion with Joint Optimization for Defending Multimodal Backdoored Models
Indranil Sur, Karan Sikka, Matthew Walmer +5
We present a Multimodal Backdoor Defense technique TIJO (Trigger Inversion using Joint Optimization). Recent work arXiv:2112.07668 has demonstrated successful backdoor attacks on m…
Probing Conceptual Understanding of Large Visual-Language Models
Madeline Schiappa, Raiyaan Abdullah, Shehreen Azad +4
In recent years large visual-language (V+L) models have achieved great success in various downstream tasks. However, it is not well studied whether these models have a conceptual g…
Detecting out-of-context objects using contextual cues
Manoj Acharya, Anirban Roy, Kaushik Koneripalli +3
This paper presents an approach to detect out-of-context (OOC) objects in an image. Given an image with a set of objects, our goal is to determine if an object is inconsistent with…
Challenges in Procedural Multimodal Machine Comprehension:A Novel Way To Benchmark
Pritish Sahu, Karan Sikka, Ajay Divakaran
We focus on Multimodal Machine Reading Comprehension (M3C) where a model is expected to answer questions based on given passage (or context), and the context and the questions can…
Modular Adaptation for Cross-Domain Few-Shot Learning
Xiao Lin, Meng Ye, Yunye Gong +4
Adapting pre-trained representations has become the go-to recipe for learning new downstream tasks with limited examples. While literature has demonstrated great successes via repr…
Generating and Evaluating Explanations of Attended and Error-Inducing Input Regions for VQA Models
Arijit Ray, Michael Cogswell, Xiao Lin +4
Attention maps, a popular heatmap-based explanation method for Visual Question Answering (VQA), are supposed to help users understand the model by highlighting portions of the imag…
Zero-Shot Learning with Knowledge Enhanced Visual Semantic Embeddings
Karan Sikka, Jihua Huang, Andrew Silberfarb +6
We improve zero-shot learning (ZSL) by incorporating common-sense knowledge in DNNs. We propose Common-Sense based Neuro-Symbolic Loss (CSNL) that formulates prior knowledge as nov…
Hybrid Consistency Training with Prototype Adaptation for Few-Shot Learning
Meng Ye, Xiao Lin, Giedrius Burachas +2
Few-Shot Learning (FSL) aims to improve a model's generalization capability in low data regimes. Recent FSL works have made steady progress via metric learning, meta learning, repr…
Deep Adaptive Semantic Logic (DASL): Compiling Declarative Knowledge into Deep Neural Networks
Karan Sikka, Andrew Silberfarb, John Byrnes +4
We introduce Deep Adaptive Semantic Logic (DASL), a novel framework for automating the generation of deep neural networks that incorporates user-provided formal knowledge to improv…
Sunny and Dark Outside?! Improving Answer Consistency in VQA through Entailed Question Generation
Arijit Ray, Karan Sikka, Ajay Divakaran +2
While models for Visual Question Answering (VQA) have steadily improved over the years, interacting with one quickly reveals that these models lack consistency. For instance, if a…
FoodX-251: A Dataset for Fine-grained Food Classification
Parneet Kaur, Karan Sikka, Weijun Wang +2
Food classification is a challenging problem due to the large number of categories, high visual similarity between different foods, as well as the lack of datasets for training sta…
Integrating Text and Image: Determining Multimodal Document Intent in Instagram Posts
Julia Kruk, Jonah Lubin, Karan Sikka +3
Computing author intent from multimodal data like Instagram posts requires modeling a complex relationship between text and image. For example, a caption might evoke an ironic cont…
Align2Ground: Weakly Supervised Phrase Grounding Guided by Image-Caption Alignment
Samyak Datta, Karan Sikka, Anirban Roy +3
We address the problem of grounding free-form textual phrases by using weak supervision from image-caption pairs. We propose a novel end-to-end model that uses caption-to-image ret…
Stacked Spatio-Temporal Graph Convolutional Networks for Action Segmentation
Pallabi Ghosh, Yi Yao, Larry S. Davis +1
We propose novel Stacked Spatio-Temporal Graph Convolutional Networks (Stacked-STGCN) for action segmentation, i.e., predicting and localizing a sequence of actions over long video…
Understanding Visual Ads by Aligning Symbols and Objects using Co-Attention
Karuna Ahuja, Karan Sikka, Anirban Roy +1
We tackle the problem of understanding visual ads where given an ad image, our goal is to rank appropriate human generated statements describing the purpose of the ad. This problem…
Zero-Shot Object Detection
Ankan Bansal, Karan Sikka, Gaurav Sharma +2
We introduce and tackle the problem of zero-shot object detection (ZSD), which aims to detect object classes which are not observed during training. We work with a challenging set…
Combining Weakly and Webly Supervised Learning for Classifying Food Images
Parneet Kaur, Karan Sikka, Ajay Divakaran
Food classification from images is a fine-grained classification problem. Manual curation of food images is cost, time and scalability prohibitive. On the other hand, web data is a…