242 citations · 1.1k across the 24 of their papers we have counts for
25 papers
ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoders
Sanghyun Woo, Shoubhik Debnath, Ronghang Hu +4
Driven by improved architectures and better representation learning frameworks, the field of visual recognition has enjoyed rapid modernization and performance boost in the early 2…
UniT3D: A Unified Transformer for 3D Dense Captioning and Visual Grounding
Dave Zhenyu Chen, Ronghang Hu, Xinlei Chen +2
Performing 3D dense captioning and visual grounding requires a common and shared understanding of the underlying multimodal relationships. However, despite some previous attempts o…
Scaling Language-Image Pre-training via Masking
Yanghao Li, Haoqi Fan, Ronghang Hu +2
We present Fast Language-Image Pre-training (FLIP), a simple and more efficient method for training CLIP. Our method randomly masks out and removes a large portion of image patches…
Exploring Long-Sequence Masked Autoencoders
Ronghang Hu, Shoubhik Debnath, Saining Xie +1
Masked Autoencoding (MAE) has emerged as an effective approach for pre-training representations across multiple domains. In contrast to discrete tokens in natural languages, the in…
FLAVA: A Foundational Language And Vision Alignment Model
Amanpreet Singh, Ronghang Hu, Vedanuj Goswami +4
State-of-the-art vision and vision-and-language models rely on large-scale visio-linguistic pretraining for obtaining good performance on a variety of downstream tasks. Generally,…
UniT: Multimodal Multitask Learning with a Unified Transformer
Ronghang Hu, Amanpreet Singh
We propose UniT, a Unified Transformer model to simultaneously learn the most prominent tasks across different domains, ranging from object detection to natural language understand…
Worldsheet: Wrapping the World in a 3D Sheet for View Synthesis from a Single Image
Ronghang Hu, Nikhila Ravi, Alexander C. Berg +1
We present Worldsheet, a method for novel view synthesis using just a single RGB image as input. The main insight is that simply shrink-wrapping a planar mesh sheet onto the input…
TextCaps: a Dataset for Image Captioning with Reading Comprehension
Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach +1
Image descriptions can help visually impaired people to quickly understand the image content. While we made significant progress in automatically describing images and optical char…
Iterative Answer Prediction with Pointer-Augmented Multimodal Transformers for TextVQA
Ronghang Hu, Amanpreet Singh, Trevor Darrell +1
Many visual scenes contain text that carries crucial information, and it is thus essential to understand text in images for downstream reasoning tasks. For example, a deep water la…
Are You Looking? Grounding to Multiple Modalities in Vision-and-Language Navigation
Ronghang Hu, Daniel Fried, Anna Rohrbach +3
Vision-and-Language Navigation (VLN) requires grounding instructions, such as "turn right and stop at the door", to routes in a visual environment. The actual grounding can connect…
Language-Conditioned Graph Networks for Relational Reasoning
Ronghang Hu, Anna Rohrbach, Trevor Darrell +1
Solving grounded language tasks often requires reasoning about relationships between objects in the context of a given task. For example, to answer the question "What color is the…
Grounding Visual Explanations
Lisa Anne Hendricks, Ronghang Hu, Trevor Darrell +1
Existing visual explanation generating agents learn to fluently justify a class prediction. However, they may mention visual attributes which reflect a strong class prior, although…
Explainable Neural Computation via Stack Neural Module Networks
Ronghang Hu, Jacob Andreas, Trevor Darrell +1
In complex inferential tasks like question answering, machine learning models must confront two challenges: the need to implement a compositional reasoning process, and, in many ap…
Generating Counterfactual Explanations with Natural Language
Lisa Anne Hendricks, Ronghang Hu, Trevor Darrell +1
Natural language explanations of deep neural network decisions provide an intuitive way for a AI agent to articulate a reasoning process. Current textual explanations learn to disc…
Speaker-Follower Models for Vision-and-Language Navigation
Daniel Fried, Ronghang Hu, Volkan Cirik +7
Navigation guided by natural language instructions presents a challenging reasoning problem for instruction followers. Natural language instructions typically identify only a few h…
Grounding Visual Explanations (Extended Abstract)
Lisa Anne Hendricks, Ronghang Hu, Trevor Darrell +1
Existing models which generate textual explanations enforce task relevance through a discriminative term loss function, but such mechanisms only weakly constrain mentioned object p…
Learning to Segment Every Thing
Ronghang Hu, Piotr Dollár, Kaiming He +2
Most methods for object instance segmentation require all training examples to be labeled with segmentation masks. This requirement makes it expensive to annotate new categories an…
Learning to Reason: End-to-End Module Networks for Visual Question Answering
Ronghang Hu, Jacob Andreas, Marcus Rohrbach +2
Natural language questions are inherently compositional, and many are most easily answered by reasoning about their decomposition into modular sub-problems. For example, to answer…
Modeling Relationships in Referential Expressions with Compositional Modular Networks
Ronghang Hu, Marcus Rohrbach, Jacob Andreas +2
People often refer to entities in an image in terms of their relationships with other entities. For example, "the black cat sitting under the table" refers to both a "black cat" en…
Utilizing Large Scale Vision and Text Datasets for Image Segmentation from Referring Expressions
Ronghang Hu, Marcus Rohrbach, Subhashini Venugopalan +1
Image segmentation from referring expressions is a joint vision and language modeling task, where the input is an image and a textual expression describing a particular region in t…
Segmentation from Natural Language Expressions
Ronghang Hu, Marcus Rohrbach, Trevor Darrell
In this paper we approach the novel problem of segmenting an image based on a natural language expression. This is different from traditional semantic segmentation over a predefine…
Natural Language Object Retrieval
Ronghang Hu, Huazhe Xu, Marcus Rohrbach +3
In this paper, we address the task of natural language object retrieval, to localize a target object within a given image based on a natural language query of the object. Natural l…
Grounding of Textual Phrases in Images by Reconstruction
Anna Rohrbach, Marcus Rohrbach, Ronghang Hu +2
Grounding (i.e. localizing) arbitrary, free-form textual phrases in visual content is a challenging problem with many applications for human-computer interaction and image-text ref…
Spatial Semantic Regularisation for Large Scale Object Detection
Damian Mrowca, Marcus Rohrbach, Judy Hoffman +3
Large scale object detection with thousands of classes introduces the problem of many contradicting false positive detections, which have to be suppressed. Class-independent non-ma…