1.1k citations · 3.7k across the 135 of their papers we have counts for
13 papers · 1 filter
ArK: Augmented Reality with Knowledge Interactive Emergent Ability
Qiuyuan Huang, Jae Sung Park, Abhinav Gupta +8
Despite the growing adoption of mixed reality and interactive AI agents, it remains challenging for these systems to generate high quality 2D/3D scenes in unseen environments. The…
Multimodal C4: An Open, Billion-scale Corpus of Images Interleaved with Text
Wanrong Zhu, Jack Hessel, Anas Awadalla +7
In-context vision and language models like Flamingo support arbitrarily interleaved sequences of images and text as input. This format not only enables few-shot learning via interl…
The Abduction of Sherlock Holmes: A Dataset for Visual Abductive Reasoning
Jack Hessel, Jena D. Hwang, Jae Sung Park +5
Humans have remarkable capacity to reason abductively and hypothesize about what lies beyond the literal content of an image. By identifying concrete visual clues scattered through…
MERLOT Reserve: Neural Script Knowledge through Vision and Language and Sound
Rowan Zellers, Jiasen Lu, Ximing Lu +7
As humans, we navigate a multimodal world, building a holistic understanding from all our senses. We introduce MERLOT Reserve, a model that represents videos jointly over time -- t…
MERLOT: Multimodal Neural Script Knowledge Models
Rowan Zellers, Ximing Lu, Jack Hessel +5
As humans, we understand events in the visual world contextually, performing multimodal reasoning across time to make inferences about the past, present, and future. We introduce M…
CLIPScore: A Reference-free Evaluation Metric for Image Captioning
Jack Hessel, Ari Holtzman, Maxwell Forbes +2
Image captioning has conventionally relied on reference-based automatic evaluations, where machine captions are compared against captions written by humans. This is in contrast to…
VinVL: Revisiting Visual Representations in Vision-Language Models
Pengchuan Zhang, Xiujun Li, Xiaowei Hu +5
This paper presents a detailed study of improving visual representations for vision language (VL) tasks and develops an improved object detection model to provide object-centric re…
VisualCOMET: Reasoning about the Dynamic Context of a Still Image
Jae Sung Park, Chandra Bhagavatula, Roozbeh Mottaghi +2
Even from a single frame of a still image, people can reason about the dynamic story of the image before, after, and beyond the frame. For example, given an image of a man struggli…
Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks
Xiujun Li, Xi Yin, Chunyuan Li +9
Large-scale pre-training methods of learning cross-modal representations on image-text pairs are becoming popular for vision-language tasks. While existing methods simply concatena…
From Recognition to Cognition: Visual Commonsense Reasoning
Rowan Zellers, Yonatan Bisk, Ali Farhadi +1
Visual understanding goes well beyond object recognition. With one glance at an image, we can effortlessly imagine the world beyond the pixels: for instance, we can infer people's…
Early Fusion for Goal Directed Robotic Vision
Aaron Walsman, Yonatan Bisk, Saadia Gabriel +4
Building perceptual systems for robotics which perform well under tight computational budgets requires novel architectures which rethink the traditional computer vision pipeline. M…
Neural Motifs: Scene Graph Parsing with Global Context
Rowan Zellers, Mark Yatskar, Sam Thomson +1
We investigate the problem of producing structured graph representations of visual scenes. Our work analyzes the role of motifs: regularly appearing substructures in scene graphs.…
Segment-Phrase Table for Semantic Segmentation, Visual Entailment and Paraphrasing
Hamid Izadinia, Fereshteh Sadeghi, Santosh Kumar Divvala +2
We introduce Segment-Phrase Table (SPT), a large collection of bijective associations between textual phrases and their corresponding segmentations. Leveraging recent progress in o…