24 citations · 77 across the 11 of their papers we have counts for
12 papers
Multimodal Unlearning Across Vision, Language, Video, and Audio: Survey of Methods, Datasets, and Benchmarks
Nobin Sarwar, Shubhashis Roy Dipta, Zheyuan Liu +1
With the growing adoption of VLMs, DMs, LLMs, and AFMs, these multimodal foundation models can inadvertently encode sensitive, copyrighted, biased, or unsafe cross-modal associatio…
Collaborating with language models for embodied reasoning
Ishita Dasgupta, Christine Kaeser-Chen, Kenneth Marino +4
Reasoning in a complex and ambiguous environment is a key goal for Reinforcement Learning (RL) agents. While some sophisticated RL agents can successfully solve difficult tasks, th…
Distilling Internet-Scale Vision-Language Models into Embodied Agents
Theodore Sumers, Kenneth Marino, Arun Ahuja +2
Instruction-following agents must ground language into their observation and action spaces. Learning to ground language is challenging, typically requiring domain-specific engineer…
Learning to Navigate Wikipedia by Taking Random Walks
Manzil Zaheer, Kenneth Marino, Will Grathwohl +7
A fundamental ability of an intelligent web-based agent is seeking out and acquiring new information. Internet search engines reliably find the correct vicinity but the top results…
A-OKVQA: A Benchmark for Visual Question Answering using World Knowledge
Dustin Schwenk, Apoorv Khandelwal, Christopher Clark +2
The Visual Question Answering (VQA) task aspires to provide a meaningful testbed for the development of AI models that can jointly reason over visual and natural language inputs. D…
KRISP: Integrating Implicit and Symbolic Knowledge for Open-Domain Knowledge-Based VQA
Kenneth Marino, Xinlei Chen, Devi Parikh +2
One of the most challenging question types in VQA is when answering the question requires outside knowledge not present in the image. In this work we study open-domain knowledge, t…
Same Object, Different Grasps: Data and Semantic Knowledge for Task-Oriented Grasping
Adithyavairavan Murali, Weiyu Liu, Kenneth Marino +2
Despite the enormous progress and generalization in robotic grasping in recent years, existing methods have yet to scale and generalize task-oriented grasping to the same extent. T…
Ask Your Humans: Using Human Instructions to Improve Generalization in Reinforcement Learning
Valerie Chen, Abhinav Gupta, Kenneth Marino
Complex, multi-task problems have proven to be difficult to solve efficiently in a sparse-reward reinforcement learning setting. In order to be sample efficient, multi-task learnin…
Empirically Verifying Hypotheses Using Reinforcement Learning
Kenneth Marino, Rob Fergus, Arthur Szlam +1
This paper formulates hypothesis verification as an RL problem. Specifically, we aim to build an agent that, given a hypothesis about the dynamics of the world, can take actions to…
OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge
Kenneth Marino, Mohammad Rastegari, Ali Farhadi +1
Visual Question Answering (VQA) in its ideal form lets us study reasoning in the joint space of vision and language and serves as a proxy for the AI task of scene understanding. Ho…
The Pose Knows: Video Forecasting by Generating Pose Futures
Jacob Walker, Kenneth Marino, Abhinav Gupta +1
Current approaches in video forecasting attempt to generate videos directly in pixel space using Generative Adversarial Networks (GANs) or Variational Autoencoders (VAEs). However,…
The More You Know: Using Knowledge Graphs for Image Classification
Kenneth Marino, Ruslan Salakhutdinov, Abhinav Gupta
One characteristic that sets humans apart from modern learning-based computer vision algorithms is the ability to acquire knowledge about the world and use that knowledge to reason…