137 citations · 494 across the 40 of their papers we have counts for
32 papers · 1 filter
3D-VisTA: Pre-trained Transformer for 3D Vision and Text Alignment
Ziyu Zhu, Xiaojian Ma, Yixin Chen +3
3D vision-language grounding (3D-VL) is an emerging field that aims to connect the 3D physical world with natural language, which is crucial for achieving embodied intelligence. Cu…
LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models
Peng Xu, Wenqi Shao, Kaipeng Zhang +7
Large Vision-Language Models (LVLMs) have recently played a dominant role in multimodal vision-language learning. Despite the great success, it lacks a holistic evaluation of their…
Prompt, Generate, then Cache: Cascade of Foundation Models makes Strong Few-shot Learners
Renrui Zhang, Xiangfei Hu, Bohao Li +5
Visual recognition in low-data regimes requires deep neural networks to learn generalized representations from limited training samples. Recently, CLIP-based methods have shown pro…
Diffusion-based Generation, Optimization, and Planning in 3D Scenes
Siyuan Huang, Zan Wang, Puhao Li +5
We introduce SceneDiffuser, a conditional generative model for 3D scene understanding. SceneDiffuser provides a unified model for solving scene-conditioned generation, optimization…
Full-Body Articulated Human-Object Interaction
Nan Jiang, Tengyu Liu, Zhexuan Cao +6
Fine-grained capturing of 3D HOI boosts human activity understanding and facilitates downstream visual tasks, including action recognition, holistic scene reconstruction, and human…
UniDA3D: Unified Domain Adaptive 3D Semantic Segmentation Pipeline
Ben Fei, Siyuan Huang, Jiakang Yuan +5
State-of-the-art 3D semantic segmentation models are trained on off-the-shelf public benchmarks, but they will inevitably face the challenge of recognition accuracy drop when these…
Perceive, Ground, Reason, and Act: A Benchmark for General-purpose Visual Representation
Jiangyong Huang, William Yicheng Zhu, Baoxiong Jia +4
Current computer vision models, unlike the human visual system, cannot yet achieve general-purpose visual understanding. Existing efforts to create a general vision model are limit…
GAPartNet: Cross-Category Domain-Generalizable Object Perception and Manipulation via Generalizable and Actionable Parts
Haoran Geng, Helin Xu, Chengyang Zhao +4
For years, researchers have been devoted to generalizable object perception and manipulation, where cross-category generalizability is highly desired yet underexplored. In this wor…
HUMANISE: Language-conditioned Human Motion Generation in 3D Scenes
Zan Wang, Yixin Chen, Tengyu Liu +3
Learning to generate diverse scene-aware and goal-oriented human motions in 3D scenes remains challenging due to the mediocre characteristics of the existing datasets on Human-Scen…
EgoTaskQA: Understanding Human Tasks in Egocentric Videos
Baoxiong Jia, Ting Lei, Song-Chun Zhu +1
Understanding human tasks through video observations is an essential capability of intelligent agents. The challenges of such capability lie in the difficulty of generating a detai…
Improving Object-centric Learning with Query Optimization
Baoxiong Jia, Yu Liu, Siyuan Huang
The ability to decompose complex natural scenes into meaningful object-centric abstractions lies at the core of human perception and reasoning. In the recent culmination of unsuper…
SQA3D: Situated Question Answering in 3D Scenes
Xiaojian Ma, Silong Yong, Zilong Zheng +4
We propose a new task to benchmark scene understanding of embodied agents: Situated Question Answering in 3D Scenes (SQA3D). Given a scene context (e.g., 3D scan), SQA3D requires t…
Infrared Invisible Clothing:Hiding from Infrared Detectors at Multiple Angles in Real World
Xiaopei Zhu, Zhanhao Hu, Siyuan Huang +2
Thermal infrared imaging is widely used in body temperature measurement, security monitoring, and so on, but its safety research attracted attention only in recent years. We propos…
Adversarial Texture for Fooling Person Detectors in the Physical World
Zhanhao Hu, Siyuan Huang, Xiaopei Zhu +3
Nowadays, cameras equipped with AI systems can capture and analyze images to detect people automatically. However, the AI system can make mistakes when receiving deliberately desig…
PartAfford: Part-level Affordance Discovery from 3D Objects
Chao Xu, Yixin Chen, He Wang +3
Understanding what objects could furnish for humans-namely, learning object affordance-is the crux to bridge perception and action. In the vision community, prior work primarily fo…
Multi-modal Sensor Fusion for Auto Driving Perception: A Survey
Keli Huang, Botian Shi, Xiang Li +3
Multi-modal fusion is a fundamental task for the perception of an autonomous driving system, which has recently intrigued many researchers. However, achieving a rather good perform…
YouRefIt: Embodied Reference Understanding with Language and Gesture
Yixin Chen, Qing Li, Deqian Kong +5
We study the understanding of embodied reference: One agent uses both language and gesture to refer to an object to another agent in a shared physical environment. Of note, this ne…
Spatio-temporal Self-Supervised Representation Learning for 3D Point Clouds
Siyuan Huang, Yichen Xie, Song-Chun Zhu +1
To date, various 3D scene understanding tasks still lack practical and generalizable pre-trained models, primarily due to the intricate nature of 3D scene understanding tasks and t…
Learning Neural Representation of Camera Pose with Matrix Representation of Pose Shift via View Synthesis
Yaxuan Zhu, Ruiqi Gao, Siyuan Huang +2
How to effectively represent camera pose is an essential problem in 3D computer vision, especially in tasks such as camera pose regression and novel view synthesis. Traditionally,…
VLGrammar: Grounded Grammar Induction of Vision and Language
Yining Hong, Qing Li, Song-Chun Zhu +1
Cognitive grammar suggests that the acquisition of language grammar is grounded within visual structures. While grammar is an essential representation of natural language, it also…
LEMMA: A Multi-view Dataset for Learning Multi-agent Multi-task Activities
Baoxiong Jia, Yixin Chen, Siyuan Huang +2
Understanding and interpreting human actions is a long-standing challenge and a critical indicator of perception in artificial intelligence. However, a few imperative components of…
A Competence-aware Curriculum for Visual Concepts Learning via Question Answering
Qing Li, Siyuan Huang, Yining Hong +1
Humans can progressively learn visual concepts from easy to hard questions. To mimic this efficient learning ability, we propose a competence-aware curriculum for visual concept le…
PerspectiveNet: 3D Object Detection from a Single RGB Image via Perspective Points
Siyuan Huang, Yixin Chen, Tao Yuan +3
Detecting 3D objects from a single RGB image is intrinsically ambiguous, thus requiring appropriate prior knowledge and intermediate representations as constraints to reduce the un…
Understanding Human Gaze Communication by Spatio-Temporal Graph Reasoning
Lifeng Fan, Wenguan Wang, Siyuan Huang +2
This paper addresses a new problem of understanding human gaze communication in social videos from both atomic-level and event-level, which is significant for studying human social…