137 citations · 494 across the 40 of their papers we have counts for
41 papers
3D-VisTA: Pre-trained Transformer for 3D Vision and Text Alignment
Ziyu Zhu, Xiaojian Ma, Yixin Chen +3
3D vision-language grounding (3D-VL) is an emerging field that aims to connect the 3D physical world with natural language, which is crucial for achieving embodied intelligence. Cu…
LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models
Peng Xu, Wenqi Shao, Kaipeng Zhang +7
Large Vision-Language Models (LVLMs) have recently played a dominant role in multimodal vision-language learning. Despite the great success, it lacks a holistic evaluation of their…
Prompt, Generate, then Cache: Cascade of Foundation Models makes Strong Few-shot Learners
Renrui Zhang, Xiangfei Hu, Bohao Li +5
Visual recognition in low-data regimes requires deep neural networks to learn generalized representations from limited training samples. Recently, CLIP-based methods have shown pro…
Diffusion-based Generation, Optimization, and Planning in 3D Scenes
Siyuan Huang, Zan Wang, Puhao Li +5
We introduce SceneDiffuser, a conditional generative model for 3D scene understanding. SceneDiffuser provides a unified model for solving scene-conditioned generation, optimization…
Full-Body Articulated Human-Object Interaction
Nan Jiang, Tengyu Liu, Zhexuan Cao +6
Fine-grained capturing of 3D HOI boosts human activity understanding and facilitates downstream visual tasks, including action recognition, holistic scene reconstruction, and human…
UniDA3D: Unified Domain Adaptive 3D Semantic Segmentation Pipeline
Ben Fei, Siyuan Huang, Jiakang Yuan +5
State-of-the-art 3D semantic segmentation models are trained on off-the-shelf public benchmarks, but they will inevitably face the challenge of recognition accuracy drop when these…
Perceive, Ground, Reason, and Act: A Benchmark for General-purpose Visual Representation
Jiangyong Huang, William Yicheng Zhu, Baoxiong Jia +4
Current computer vision models, unlike the human visual system, cannot yet achieve general-purpose visual understanding. Existing efforts to create a general vision model are limit…
GAPartNet: Cross-Category Domain-Generalizable Object Perception and Manipulation via Generalizable and Actionable Parts
Haoran Geng, Helin Xu, Chengyang Zhao +4
For years, researchers have been devoted to generalizable object perception and manipulation, where cross-category generalizability is highly desired yet underexplored. In this wor…
HUMANISE: Language-conditioned Human Motion Generation in 3D Scenes
Zan Wang, Yixin Chen, Tengyu Liu +3
Learning to generate diverse scene-aware and goal-oriented human motions in 3D scenes remains challenging due to the mediocre characteristics of the existing datasets on Human-Scen…
EgoTaskQA: Understanding Human Tasks in Egocentric Videos
Baoxiong Jia, Ting Lei, Song-Chun Zhu +1
Understanding human tasks through video observations is an essential capability of intelligent agents. The challenges of such capability lie in the difficulty of generating a detai…
Improving Object-centric Learning with Query Optimization
Baoxiong Jia, Yu Liu, Siyuan Huang
The ability to decompose complex natural scenes into meaningful object-centric abstractions lies at the core of human perception and reasoning. In the recent culmination of unsuper…
SQA3D: Situated Question Answering in 3D Scenes
Xiaojian Ma, Silong Yong, Zilong Zheng +4
We propose a new task to benchmark scene understanding of embodied agents: Situated Question Answering in 3D Scenes (SQA3D). Given a scene context (e.g., 3D scan), SQA3D requires t…
Neural-Symbolic Recursive Machine for Systematic Generalization
Qing Li, Yixin Zhu, Yitao Liang +3
Current learning models often struggle with human-like systematic generalization, particularly in learning compositional rules from limited data and extrapolating them to novel com…
GenDexGrasp: Generalizable Dexterous Grasping
Puhao Li, Tengyu Liu, Yuyang Li +4
Generating dexterous grasping has been a long-standing and challenging robotic task. Despite recent progress, existing methods primarily suffer from two issues. First, most prior a…
Infrared Invisible Clothing:Hiding from Infrared Detectors at Multiple Angles in Real World
Xiaopei Zhu, Zhanhao Hu, Siyuan Huang +2
Thermal infrared imaging is widely used in body temperature measurement, security monitoring, and so on, but its safety research attracted attention only in recent years. We propos…
Adversarial Texture for Fooling Person Detectors in the Physical World
Zhanhao Hu, Siyuan Huang, Xiaopei Zhu +3
Nowadays, cameras equipped with AI systems can capture and analyze images to detect people automatically. However, the AI system can make mistakes when receiving deliberately desig…
PartAfford: Part-level Affordance Discovery from 3D Objects
Chao Xu, Yixin Chen, He Wang +3
Understanding what objects could furnish for humans-namely, learning object affordance-is the crux to bridge perception and action. In the vision community, prior work primarily fo…
Multi-modal Sensor Fusion for Auto Driving Perception: A Survey
Keli Huang, Botian Shi, Xiang Li +3
Multi-modal fusion is a fundamental task for the perception of an autonomous driving system, which has recently intrigued many researchers. However, achieving a rather good perform…
YouRefIt: Embodied Reference Understanding with Language and Gesture
Yixin Chen, Qing Li, Deqian Kong +5
We study the understanding of embodied reference: One agent uses both language and gesture to refer to an object to another agent in a shared physical environment. Of note, this ne…
Spatio-temporal Self-Supervised Representation Learning for 3D Point Clouds
Siyuan Huang, Yichen Xie, Song-Chun Zhu +1
To date, various 3D scene understanding tasks still lack practical and generalizable pre-trained models, primarily due to the intricate nature of 3D scene understanding tasks and t…
Learning Neural Representation of Camera Pose with Matrix Representation of Pose Shift via View Synthesis
Yaxuan Zhu, Ruiqi Gao, Siyuan Huang +2
How to effectively represent camera pose is an essential problem in 3D computer vision, especially in tasks such as camera pose regression and novel view synthesis. Traditionally,…
Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoning
Pan Lu, Ran Gong, Shibiao Jiang +4
Geometry problem solving has attracted much attention in the NLP community recently. The task is challenging as it requires abstract problem understanding and symbolic reasoning wi…
VLGrammar: Grounded Grammar Induction of Vision and Language
Yining Hong, Qing Li, Song-Chun Zhu +1
Cognitive grammar suggests that the acquisition of language grammar is grounded within visual structures. While grammar is an essential representation of natural language, it also…
A Minimalist Dataset for Systematic Generalization of Perception, Syntax, and Semantics
Qing Li, Siyuan Huang, Yining Hong +3
Inspired by humans' exceptional ability to master arithmetic and generalize to new problems, we present a new dataset, Handwritten arithmetic with INTegers (HINT), to examine machi…