312 citations · 2.2k across the 74 of their papers we have counts for
24 papers · 1 filter
Doduo: Learning Dense Visual Correspondence from Unsupervised Semantic-Aware Flow
Zhenyu Jiang, Hanwen Jiang, Yuke Zhu
Dense visual correspondence plays a vital role in robotic perception. This work focuses on establishing the dense correspondence between a pair of images that captures dynamic scen…
Fast Monocular Scene Reconstruction with Global-Sparse Local-Dense Grids
Wei Dong, Chris Choy, Charles Loop +3
Indoor scene reconstruction from monocular images has long been sought after by augmented reality and robotics developers. Recent advances in neural field representations and monoc…
Re-ViLM: Retrieval-Augmented Visual Language Model for Zero and Few-Shot Image Captioning
Zhuolin Yang, Wei Ping, Zihan Liu +13
Augmenting pretrained language models (LMs) with a vision encoder (e.g., Flamingo) has obtained the state-of-the-art results in image-to-text generation. However, these models stor…
Few-View Object Reconstruction with Unknown Categories and Camera Poses
Hanwen Jiang, Zhenyu Jiang, Kristen Grauman +1
While object reconstruction has made great strides in recent years, current methods typically require densely captured images and/or known camera poses, and generalize poorly to no…
COOPERNAUT: End-to-End Driving with Cooperative Perception for Networked Vehicles
Jiaxun Cui, Hang Qiu, Dian Chen +2
Optical sensors and learning algorithms for autonomous vehicles have dramatically advanced in the past few years. Nonetheless, the reliability of today's autonomous vehicles is hin…
Bongard-HOI: Benchmarking Few-Shot Visual Reasoning for Human-Object Interactions
Huaizu Jiang, Xiaojian Ma, Weili Nie +4
A significant gap remains between today's visual pattern recognition models and human-level visual cognition especially when it comes to few-shot learning and compositional reasoni…
RelViT: Concept-guided Vision Transformer for Visual Relational Reasoning
Xiaojian Ma, Weili Nie, Zhiding Yu +5
Reasoning about visual relationships is central to how humans interpret the visual world. This task remains challenging for current deep learning algorithms since it requires addre…
ACID: Action-Conditional Implicit Visual Dynamics for Deformable Object Manipulation
Bokui Shen, Zhenyu Jiang, Christopher Choy +4
Manipulating volumetric deformable objects in the real world, like plush toys and pizza dough, bring substantial challenges due to infinite shape variations, non-rigid motions, and…
Ditto: Building Digital Twins of Articulated Objects from Interaction
Zhenyu Jiang, Cheng-Chun Hsu, Yuke Zhu
Digitizing physical objects into the virtual world has the potential to unlock new research and applications in embodied AI and mixed reality. This work focuses on recreating inter…
DiscoBox: Weakly Supervised Instance Segmentation and Semantic Correspondence from Box Supervision
Shiyi Lan, Zhiding Yu, Christopher Choy +5
We introduce DiscoBox, a novel framework that jointly learns instance segmentation and semantic correspondence using bounding box supervision. Specifically, we propose a self-ensem…
Dynamic Metric Learning: Towards a Scalable Metric Space to Accommodate Multiple Semantic Scales
Yifan Sun, Yuke Zhu, Yuhan Zhang +4
This paper introduces a new fundamental characteristic, \ie, the dynamic range, from real-world metric tools to deep visual recognition. In metrology, the dynamic range is a basic…
Spherical Feature Transform for Deep Metric Learning
Yuke Zhu, Yan Bai, Yichen Wei
Data augmentation in feature space is effective to increase data diversity. Previous methods assume that different classes have the same covariance in their feature distributions.…
6-PACK: Category-level 6D Pose Tracker with Anchor-Based Keypoints
Chen Wang, Roberto Martín-Martín, Danfei Xu +5
We present 6-PACK, a deep learning approach to category-level 6D object pose tracking on RGB-D data. Our method tracks in real-time novel object instances of known object categorie…
Situational Fusion of Visual Representation for Visual Navigation
Bokui Shen, Danfei Xu, Yuke Zhu +3
A complex visual navigation task puts an agent in different situations which call for a diverse range of visual perception abilities. For example, to "go to the nearest chair", the…
DenseFusion: 6D Object Pose Estimation by Iterative Dense Fusion
Chen Wang, Danfei Xu, Yuke Zhu +4
A key technical challenge in performing 6D object pose estimation from RGB-D image is to fully leverage the two complementary data sources. Prior works either extract information f…
Neural Task Graphs: Generalizing to Unseen Tasks from a Single Video Demonstration
De-An Huang, Suraj Nair, Danfei Xu +5
Our goal is to generate a policy to complete an unseen task given just a single video demonstration of the task in a given domain. We hypothesize that to successfully generalize to…
AI2-THOR: An Interactive 3D Environment for Visual AI
Eric Kolve, Roozbeh Mottaghi, Winson Han +10
We introduce The House Of inteRactions (THOR), a framework for visual AI research, available at http://ai2thor.allenai.org. AI2-THOR consists of near photo-realistic 3D indoor scen…
Visual Semantic Planning using Deep Successor Representations
Yuke Zhu, Daniel Gordon, Eric Kolve +5
A crucial capability of real-world intelligent agents is their ability to plan a sequence of actions to achieve their goals in the visual world. In this work, we address the proble…
Scene Graph Generation by Iterative Message Passing
Danfei Xu, Yuke Zhu, Christopher B. Choy +1
Understanding a visual scene goes beyond recognizing individual objects in isolation. Relationships between objects also constitute rich semantic information about the scene. In th…
Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learning
Yuke Zhu, Roozbeh Mottaghi, Eric Kolve +4
Two less addressed issues of deep reinforcement learning are (1) lack of generalization capability to new target goals, and (2) data inefficiency i.e., the model requires several (…
Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations
Ranjay Krishna, Yuke Zhu, Oliver Groth +9
Despite progress in perceptual tasks such as image classification, computers still perform poorly on cognitive tasks such as image description and question answering. Cognition is…
Visual7W: Grounded Question Answering in Images
Yuke Zhu, Oliver Groth, Michael Bernstein +1
We have seen great progress in basic perceptual tasks such as object recognition and detection. However, AI models still fail to match humans in high-level vision tasks due to the…
Action Recognition by Hierarchical Mid-level Action Elements
Tian Lan, Yuke Zhu, Amir Roshan Zamir +1
Realistic videos of human actions exhibit rich spatiotemporal structures at multiple levels of granularity: an action can always be decomposed into multiple finer-grained elements…
Building a Large-scale Multimodal Knowledge Base System for Answering Visual Queries
Yuke Zhu, Ce Zhang, Christopher Ré +1
The complexity of the visual world creates significant challenges for comprehensive visual understanding. In spite of recent successes in visual recognition, today's vision systems…