940 citations · 2.5k across the 35 of their papers we have counts for
35 papers
MOMA-Force: Visual-Force Imitation for Real-World Mobile Manipulation
Taozheng Yang, Ya Jing, Hongtao Wu +5
In this paper, we present a novel method for mobile manipulators to perform multiple contact-rich manipulation tasks. While learning-based methods have the potential to generate ac…
Exploring Visual Pre-training for Robot Manipulation: Datasets, Models and Methods
Ya Jing, Xuelin Zhu, Xingbin Liu +4
Visual pre-training with large-scale real-world data has made great progress in recent years, showing great potential in robot learning with pixel observations. However, the recipe…
What Matters in Training a GPT4-Style Language Model with Multimodal Inputs?
Yan Zeng, Hanbo Zhang, Jiani Zheng +5
Recent advancements in Large Language Models (LLMs) such as GPT4 have displayed exceptional multi-modal capabilities in following open-ended instructions given images. However, the…
ClickSeg: 3D Instance Segmentation with Click-Level Weak Annotations
Leyao Liu, Tao Kong, Minzhao Zhu +2
3D instance segmentation methods often require fully-annotated dense labels for training, which are costly to obtain. In this paper, we present ClickSeg, a novel click-level weakly…
Learning to Explore Informative Trajectories and Samples for Embodied Perception
Ya Jing, Tao Kong
We are witnessing significant progress on perception models, specifically those trained on large-scale internet images. However, efficiently generalizing these perception models to…
Towards Unifying Reference Expression Generation and Comprehension
Duo Zheng, Tao Kong, Ya Jing +2
Reference Expression Generation (REG) and Comprehension (REC) are two highly correlated tasks. Modeling REG and REC simultaneously for utilizing the relation between them is a prom…
Generative Category-Level Shape and Pose Estimation with Semantic Primitives
Guanglin Li, Yifeng Li, Zhichao Ye +4
Empowering autonomous agents with 3D understanding for daily objects is a grand challenge in robotics applications. When exploring in an unknown environment, existing methods for o…
Exploring Target Representations for Masked Autoencoders
Xingbin Liu, Jinghao Zhou, Tao Kong +2
Masked autoencoders have become popular training paradigms for self-supervised visual representation learning. These models randomly mask a portion of the input and reconstruct the…
3D Part Assembly Generation with Instance Encoded Transformer
Rufeng Zhang, Tao Kong, Weihao Wang +2
It is desirable to enable robots capable of automatic assembly. Structural understanding of object parts plays a crucial role in this task yet remains relatively unexplored. In thi…
Learning Design and Construction with Varying-Sized Materials via Prioritized Memory Resets
Yunfei Li, Tao Kong, Lei Li +1
Can a robot autonomously learn to design and construct a bridge from varying-sized blocks without a blueprint? It is a challenging task with long horizon and sparse reward -- the r…
Navigating to Objects in Unseen Environments by Distance Prediction
Minzhao Zhu, Binglei Zhao, Tao Kong
Object Goal Navigation (ObjectNav) task is to navigate an agent to an object category in unseen environments without a pre-built map. In this paper, we solve this task by predictin…
iBOT: Image BERT Pre-Training with Online Tokenizer
Jinghao Zhou, Chen Wei, Huiyu Wang +4
The success of language Transformers is primarily attributed to the pretext task of masked language modeling (MLM), where texts are first tokenized into semantically meaningful pie…
Self-Supervised Learning by Estimating Twin Class Distributions
Feng Wang, Tao Kong, Rufeng Zhang +2
We present TWIST, a simple and theoretically explainable self-supervised representation learning method by classifying large-scale unlabeled datasets in an end-to-end way. We emplo…
Simultaneous Semantic and Collision Learning for 6-DoF Grasp Pose Estimation
Yiming Li, Tao Kong, Ruihang Chu +3
Grasping in cluttered scenes has always been a great challenge for robots, due to the requirement of the ability to well understand the scene and object information. Previous works…
ICM-3D: Instantiated Category Modeling for 3D Instance Segmentation
Ruihang Chu, Yukang Chen, Tao Kong +2
Separating 3D point clouds into individual instances is an important task for 3D vision. It is challenging due to the unknown and varying number of instances in a scene. Existing d…
Learning to Design and Construct Bridge without Blueprint
Yunfei Li, Tao Kong, Lei Li +2
Autonomous assembly has been a desired functionality of many intelligent robot systems. We study a new challenging assembly task, designing and constructing a bridge without a blue…
SOLO: A Simple Framework for Instance Segmentation
Xinlong Wang, Rufeng Zhang, Chunhua Shen +2
Compared to many other dense prediction tasks, e.g., semantic segmentation, it is the arbitrary number of instances that has made instance segmentation much more challenging. In or…
Adversarial Option-Aware Hierarchical Imitation Learning
Mingxuan Jing, Wenbing Huang, Fuchun Sun +4
It has been a challenge to learning skills for an agent from long-horizon unannotated demonstrations. Existing approaches like Hierarchical Imitation Learning(HIL) are prone to com…
Scale-aware Automatic Augmentation for Object Detection
Yukang Chen, Yanwei Li, Tao Kong +4
We propose Scale-aware AutoAug to learn data augmentation policies for object detection. We define a new scale-aware search space, where both image- and box-level augmentations are…
Locate then Segment: A Strong Pipeline for Referring Image Segmentation
Ya Jing, Tao Kong, Wei Wang +3
Referring image segmentation aims to segment the objects referred by a natural language expression. Previous methods usually focus on designing an implicit and recurrent feature in…
TransTrack: Multiple Object Tracking with Transformer
Peize Sun, Jinkun Cao, Yi Jiang +5
In this work, we propose TransTrack, a simple but efficient scheme to solve the multiple object tracking problems. TransTrack leverages the transformer architecture, which is an at…
Sparse R-CNN: End-to-End Object Detection with Learnable Proposals
Peize Sun, Rufeng Zhang, Yi Jiang +8
We present Sparse R-CNN, a purely sparse method for object detection in images. Existing works on object detection heavily rely on dense object candidates, such as anchor boxes…
Dense Contrastive Learning for Self-Supervised Visual Pre-Training
Xinlong Wang, Rufeng Zhang, Chunhua Shen +2
To date, most existing self-supervised learning methods are designed and optimized for image classification. These pre-trained models can be sub-optimal for dense prediction tasks…
SOLOv2: Dynamic and Fast Instance Segmentation
Xinlong Wang, Rufeng Zhang, Tao Kong +2
In this work, we aim at building a simple, direct, and fast instance segmentation framework with strong performance. We follow the principle of the SOLO method of Wang et al. "SOLO…