352 citations · 568 across the 21 of their papers we have counts for
21 papers
Learning Panorama-Aware VLA for Mobile Manipulation with Whole-Body Teleoperation
Donglin Yang, Haoran Chen, Xingyu Chen +6
Mobile manipulation is a key capability for embodied intelligence, enabling robots to accomplish complex multi-stage tasks in open-world environments. However, mobile manipulation…
3D-VisTA: Pre-trained Transformer for 3D Vision and Text Alignment
Ziyu Zhu, Xiaojian Ma, Yixin Chen +3
3D vision-language grounding (3D-VL) is an emerging field that aims to connect the 3D physical world with natural language, which is crucial for achieving embodied intelligence. Cu…
Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agents
Zihao Wang, Shaofei Cai, Guanzhou Chen +3
We investigate the challenge of task planning for multi-task embodied agents in open-world environments. Two main difficulties are identified: 1) executing plans in an open-world e…
Open-World Multi-Task Control Through Goal-Aware Representation Learning and Adaptive Horizon Prediction
Shaofei Cai, Zihao Wang, Xiaojian Ma +2
We study the problem of learning goal-conditioned policies in Minecraft, a popular, widely accessible yet challenging open-ended environment for developing human-level multi-task a…
Perceive, Ground, Reason, and Act: A Benchmark for General-purpose Visual Representation
Jiangyong Huang, William Yicheng Zhu, Baoxiong Jia +4
Current computer vision models, unlike the human visual system, cannot yet achieve general-purpose visual understanding. Existing efforts to create a general vision model are limit…
SQA3D: Situated Question Answering in 3D Scenes
Xiaojian Ma, Silong Yong, Zilong Zheng +4
We propose a new task to benchmark scene understanding of embodied agents: Situated Question Answering in 3D Scenes (SQA3D). Given a scene context (e.g., 3D scan), SQA3D requires t…
Latent Diffusion Energy-Based Model for Interpretable Text Modeling
Peiyu Yu, Sirui Xie, Xiaojian Ma +6
Latent space Energy-Based Models (EBMs), also known as energy-based priors, have drawn growing interests in generative modeling. Fueled by its flexibility in the formulation and st…
Bongard-HOI: Benchmarking Few-Shot Visual Reasoning for Human-Object Interactions
Huaizu Jiang, Xiaojian Ma, Weili Nie +4
A significant gap remains between today's visual pattern recognition models and human-level visual cognition especially when it comes to few-shot learning and compositional reasoni…
RelViT: Concept-guided Vision Transformer for Visual Relational Reasoning
Xiaojian Ma, Weili Nie, Zhiding Yu +5
Reasoning about visual relationships is central to how humans interpret the visual world. This task remains challenging for current deep learning algorithms since it requires addre…
Unsupervised Foreground Extraction via Deep Region Competition
Peiyu Yu, Sirui Xie, Xiaojian Ma +3
We present Deep Region Competition (DRC), an algorithm designed to extract foreground objects from images in a fully unsupervised manner. Foreground extraction can be viewed as a s…
Adversarial Option-Aware Hierarchical Imitation Learning
Mingxuan Jing, Wenbing Huang, Fuchun Sun +4
It has been a challenge to learning skills for an agent from long-horizon unannotated demonstrations. Existing approaches like Hierarchical Imitation Learning(HIL) are prone to com…
HALMA: Humanlike Abstraction Learning Meets Affordance in Rapid Problem Solving
Sirui Xie, Xiaojian Ma, Peiyu Yu +3
Humans learn compositional and causal abstraction, \ie, knowledge, in response to the structure of naturalistic tasks. When presented with a problem-solving task involving some obj…
A Mobile Robot Hand-Arm Teleoperation System by Vision and IMU
Shuang Li, Jiaxi Jiang, Philipp Ruppel +5
In this paper, we present a multimodal mobile teleoperation system that consists of a novel vision-based hand pose regression network (Transteleop) and an IMU-based arm tracking me…
Robust Robotic Pouring using Audition and Haptics
Hongzhuo Liang, Chuangchuang Zhou, Shuang Li +6
Robust and accurate estimation of liquid height lies as an essential part of pouring tasks for service robots. However, vision-based methods often fail in occluded conditions while…
Theory-based Causal Transfer: Integrating Instance-level Induction and Abstract-level Structure Learning
Mark Edmonds, Xiaojian Ma, Siyuan Qi +3
Learning transferable knowledge across similar but different settings is a fundamental component of generalized intelligence. In this paper, we approach the transfer learning chall…
Reinforcement Learning from Imperfect Demonstrations under Soft Expert Guidance
Mingxuan Jing, Xiaojian Ma, Wenbing Huang +4
In this paper, we study Reinforcement Learning from Demonstrations (RLfD) that improves the exploration efficiency of Reinforcement Learning (RL) by providing expert demonstrations…
Imitation Learning from Observations by Minimizing Inverse Dynamics Disagreement
Chao Yang, Xiaojian Ma, Wenbing Huang +4
This paper studies Learning from Observations (LfO) for imitation learning with access to state-only demonstrations. In contrast to Learning from Demonstration (LfD) that involves…
Making Sense of Audio Vibration for Liquid Height Estimation in Robotic Pouring
Hongzhuo Liang, Shuang Li, Xiaojian Ma +4
In this paper, we focus on the challenging perception problem in robotic pouring. Most of the existing approaches either leverage visual or haptic information. However, these techn…
Vision-based Teleoperation of Shadow Dexterous Hand using End-to-End Deep Neural Network
Shuang Li, Xiaojian Ma, Hongzhuo Liang +5
In this paper, we present TeachNet, a novel neural network architecture for intuitive and markerless vision-based teleoperation of dexterous robotic hands. Robot joint angles are d…
PointNetGPD: Detecting Grasp Configurations from Point Sets
Hongzhuo Liang, Xiaojian Ma, Shuang Li +5
In this paper, we propose an end-to-end grasp evaluation model to address the challenging problem of localizing robot grasp configurations directly from the point cloud. Compared t…
Learning and Inferring Movement with Deep Generative Model
Mingxuan Jing, Xiaojian Ma, Fuchun Sun +1
Learning and inference movement is a very challenging problem due to its high dimensionality and dependency to varied environments or tasks. In this paper, we propose an effective…