2.4k citations · 5.2k across the 67 of their papers we have counts for
30 papers · 1 filter
RIC: Rotate-Inpaint-Complete for Generalizable Scene Reconstruction
Isaac Kasahara, Shubham Agrawal, Selim Engin +3
General scene reconstruction refers to the task of estimating the full 3D geometry and texture of a scene containing previously unseen objects. In many practical applications such…
DataComp: In search of the next generation of multimodal datasets
Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang +31
Multimodal datasets are a critical component in recent breakthroughs such as Stable Diffusion and GPT-4, yet their design does not receive the same research attention as model arch…
Tracking and Reconstructing Hand Object Interactions from Point Cloud Sequences in the Wild
Jiayi Chen, Mi Yan, Jiazhao Zhang +6
In this work, we tackle the challenging task of jointly tracking hand object pose and reconstructing their shapes from depth point cloud sequences in the wild, given the initial po…
TANDEM3D: Active Tactile Exploration for 3D Object Recognition
Jingxi Xu, Han Lin, Shuran Song +1
Tactile recognition of 3D objects remains a challenging task. Compared to 2D shapes, the complex geometry of 3D surfaces requires richer tactile signals, more dexterous actions, an…
Patching open-vocabulary models by interpolating weights
Gabriel Ilharco, Mitchell Wortsman, Samir Yitzhak Gadre +5
Open-vocabulary models like CLIP achieve high accuracy across many image classification tasks. However, there are still settings where their zero-shot performance is far from optim…
Semantic Abstraction: Open-World 3D Scene Understanding from 2D Vision-Language Models
Huy Ha, Shuran Song
We study open-world 3D scene understanding, a family of tasks that require agents to reason about their 3D environment with an open-set vocabulary and out-of-domain visual inputs -…
Continuous Scene Representations for Embodied AI
Samir Yitzhak Gadre, Kiana Ehsani, Shuran Song +1
We propose Continuous Scene Representations (CSR), a scene representation constructed by an embodied agent navigating within a space, where objects and their relationships are mode…
CoWs on Pasture: Baselines and Benchmarks for Language-Driven Zero-Shot Object Navigation
Samir Yitzhak Gadre, Mitchell Wortsman, Gabriel Ilharco +2
For robots to be generally useful, they must be able to find arbitrary objects described by people (i.e., be language-driven) even without expensive navigation training on in-domai…
Leveraging SE(3) Equivariance for Self-Supervised Category-Level Object Pose Estimation
Xiaolong Li, Yijia Weng, Li Yi +4
Category-level object pose estimation aims to find 6D object poses of previously unseen object instances from known categories without access to object CAD models. To reduce the hu…
UMPNet: Universal Manipulation Policy Network for Articulated Objects
Zhenjia Xu, Zhanpeng He, Shuran Song
We introduce the Universal Manipulation Policy Network (UMPNet) -- a single image-based policy network that infers closed-loop action sequences for manipulating arbitrary articulat…
Act the Part: Learning Interaction Strategies for Articulated Object Part Discovery
Samir Yitzhak Gadre, Kiana Ehsani, Shuran Song
People often use physical intuition when manipulating articulated objects, irrespective of object semantics. Motivated by this observation, we identify an important embodied task w…
GarmentNets: Category-Level Pose Estimation for Garments via Canonical Space Shape Completion
Cheng Chi, Shuran Song
This paper tackles the task of category-level pose estimation for garments. With a near infinite degree of freedom, a garment's full configuration (i.e., poses) is often described…
SSCNav: Confidence-Aware Semantic Scene Completion for Visual Semantic Navigation
Yiqing Liang, Boyuan Chen, Shuran Song
This paper focuses on visual semantic navigation, the task of producing actions for an active agent to navigate to a specified target object category in an unknown environment. To…
Multitask Learning Strengthens Adversarial Robustness
Chengzhi Mao, Amogh Gupta, Vikram Nitin +4
Although deep networks achieve strong accuracy on a range of computer vision benchmarks, they remain vulnerable to adversarial attacks, where imperceptible input perturbations fool…
Category-Level Articulated Object Pose Estimation
Xiaolong Li, He Wang, Li Yi +3
This project addresses the task of category-level pose estimation for articulated objects from a single depth image. We present a novel category-level approach that correctly accom…
Grasping in the Wild:Learning 6DoF Closed-Loop Grasping from Low-Cost Demonstrations
Shuran Song, Andy Zeng, Johnny Lee +1
Intelligent manipulation benefits from the capacity to flexibly control an end-effector with high degrees of freedom (DoF) and dynamically react to the environment. However, due to…
ClearGrasp: 3D Shape Estimation of Transparent Objects for Manipulation
Shreeyak S. Sajjan, Matthew Moore, Mike Pan +4
Transparent objects are a common part of everyday life, yet they possess unique visual properties that make them incredibly difficult for standard 3D sensors to produce accurate de…
Neural Illumination: Lighting Prediction for Indoor Environments
Shuran Song, Thomas Funkhouser
This paper addresses the task of estimating the light arriving from all directions to a 3D point observed at a selected pixel in an RGB image. This task is challenging because it r…
Normalized Object Coordinate Space for Category-Level 6D Object Pose and Size Estimation
He Wang, Srinath Sridhar, Jingwei Huang +3
The goal of this paper is to estimate the 6D pose and dimensions of unseen object instances in an RGB-D image. Contrary to "instance-level" 6D pose estimation tasks, our problem as…
Im2Pano3D: Extrapolating 360 Structure and Semantics Beyond the Field of View
Shuran Song, Andy Zeng, Angel X. Chang +3
We present Im2Pano3D, a convolutional neural network that generates a dense prediction of 3D structure and a probability distribution of semantic labels for a full 360 panoramic vi…
Matterport3D: Learning from RGB-D Data in Indoor Environments
Angel Chang, Angela Dai, Thomas Funkhouser +6
Access to large, diverse RGB-D datasets is critical for training RGB-D scene understanding algorithms. However, existing datasets still cover only a limited number of views or a re…
Physically-Based Rendering for Indoor Scene Understanding Using Convolutional Neural Networks
Yinda Zhang, Shuran Song, Ersin Yumer +4
Indoor scene understanding is central to applications such as robot navigation and human companion assistance. Over the last years, data-driven deep neural networks have outperform…
Semantic Scene Completion from a Single Depth Image
Shuran Song, Fisher Yu, Andy Zeng +3
This paper focuses on semantic scene completion, a task for producing a complete 3D voxel representation of volumetric occupancy and semantic labels for a scene from a single-view…
Multi-view Self-supervised Deep Learning for 6D Pose Estimation in the Amazon Picking Challenge
Andy Zeng, Kuan-Ting Yu, Shuran Song +4
Robot warehouse automation has attracted significant interest in recent years, perhaps most visibly in the Amazon Picking Challenge (APC). A fully autonomous warehouse pick-and-pla…