24 papers
ViLoMan: Learning Visual-Proprioceptive Whole-Body Loco-Manipulation Skills for Humanoid Robots
Zejie Tian, Ruibing Hou, Bingpeng Ma +2
Humanoid loco-manipulation requires adaptive whole-body coordination to seamlessly integrate locomotion and physical interaction. Despite recent advances, learning autonomous loco-…
EMPIRE: Explicit Manipulation Planning as a Learnable Intermediate Representation for Egocentric Hand-Motion Forecasting
Wen Wang, Ruibing Hou, Hong Chang +2
Forecasting dexterous hand motions from egocentric observations is fundamental to intelligent interactive systems. Existing VLM-based methods typically map observations directly to…
AnyMo: Scaling Any-Modality Conditional Motion Generation with Masked Modeling
Yiheng Li, Zhuo Li, Ruibing Hou +4
Conditional human motion generation remains a fundamental challenge in computer vision and robotics. Despite significant progress, current methods are often constrained by fixed mo…
Component-Based Out-of-Distribution Detection
Wenrui Liu, Hong Chang, Ruibing Hou +2
Out-of-Distribution (OOD) detection requires sensitivity to subtle shifts without overreacting to natural In-Distribution (ID) diversity. However, from the viewpoint of detection g…
EgoMotion: Hierarchical Reasoning and Diffusion for Egocentric Vision-Language Motion Generation
Ruibing Hou, Mingyue Zhou, Yuwei Gui +5
Faithfully modeling human behavior in dynamic environments is a foundational challenge for embodied intelligence. While conditional motion synthesis has achieved significant advanc…
DreamActor-M2: Universal Character Image Animation via Spatiotemporal In-Context Learning
Mingshuang Luo, Shuang Liang, Zhengkun Rong +7
Character image animation aims to synthesize high-fidelity videos by transferring motion from a driving sequence to a static reference image. Despite recent advancements, existing…
CLIP-Guided Adaptable Self-Supervised Learning for Human-Centric Visual Tasks
Mingshuang Luo, Ruibing Hou, Bo Chao +4
Human-centric visual analysis plays a pivotal role in diverse applications, including surveillance, healthcare, and human-computer interaction. With the emergence of large-scale un…
Revisiting Logit Distributions for Reliable Out-of-Distribution Detection
Jiachen Liang, Ruibing Hou, Minyang Hu +3
Out-of-distribution (OOD) detection is critical for ensuring the reliability of deep learning models in open-world applications. While post-hoc methods are favored for their effici…
KnowMol: Advancing Molecular Large Language Models with Multi-Level Chemical Knowledge
Zaifei Yang, Hong Chang, Ruibing Hou +2
The molecular large language models have garnered widespread attention due to their promising potential on molecular applications. However, current molecular large language models…
Revisiting Multimodal Positional Encoding in Vision-Language Models
Jie Huang, Xuejing Liu, Sibo Song +4
Multimodal position encoding is essential for vision-language models, yet there has been little systematic investigation into multimodal position encoding. We conduct a comprehensi…
MotionVerse: A Unified Multimodal Framework for Motion Comprehension, Generation and Editing
Ruibing Hou, Mingshuang Luo, Hongyu Pan +2
This paper proposes MotionVerse, a unified framework that harnesses the capabilities of Large Language Models (LLMs) to comprehend, generate, and edit human motion in both single-p…
HumanPCR: Probing MLLM Capabilities in Diverse Human-Centric Scenes
Keliang Li, Hongze Shen, Hao Shi +9
The aspiration for artificial general intelligence, fueled by the rapid progress of multimodal models, demands human-comparable performance across diverse environments. We propose…
unCLIP: Improving CLIP's Visual Detail Capturing Ability via Inverting unCLIP
Yinqi Li, Jiahe Zhao, Hong Chang +3
Contrastive Language-Image Pre-training (CLIP) has become a foundation model and has been applied to various vision and multimodal tasks. However, recent works indicate that CLIP f…
DIVE: Inverting Conditional Diffusion Models for Discriminative Tasks
Yinqi Li, Hong Chang, Ruibing Hou +2
Diffusion models have shown remarkable progress in various generative tasks such as image and video generation. This paper studies the problem of leveraging pretrained diffusion mo…
HIS-GPT: Towards 3D Human-In-Scene Multimodal Understanding
Jiahe Zhao, Ruibing Hou, Zejie Tian +2
We propose a new task to benchmark human-in-scene understanding for embodied agents: Human-In-Scene Question Answering (HIS-QA). Given a human motion within a 3D scene, HIS-QA requ…
MATS: An Audio Language Model under Text-only Supervision
Wen Wang, Ruibing Hou, Hong Chang +2
Large audio-language models (LALMs), built upon powerful Large Language Models (LLMs), have exhibited remarkable audio comprehension and reasoning capabilities. However, the traini…
RefHCM: A Unified Model for Referring Perceptions in Human-Centric Scenarios
Jie Huang, Ruibing Hou, Jiahe Zhao +2
Human-centric perceptions play a crucial role in real-world applications. While recent human-centric works have achieved impressive progress, these efforts are often constrained to…
UMFC: Unsupervised Multi-Domain Feature Calibration for Vision-Language Models
Jiachen Liang, Ruibing Hou, Minyang Hu +3
Pre-trained vision-language models (e.g., CLIP) have shown powerful zero-shot transfer capabilities. But they still struggle with domain shifts and typically require labeled data t…
UniPose: A Unified Multimodal Framework for Human Pose Comprehension, Generation and Editing
Yiheng Li, Ruibing Hou, Hong Chang +2
Human pose plays a crucial role in the digital age. While recent works have achieved impressive progress in understanding and generating human poses, they often support only a sing…
Morph: A Motion-free Physics Optimization Framework for Human Motion Generation
Zhuo Li, Mingshuang Luo, Ruibing Hou +5
Human motion generation has been widely studied due to its crucial role in areas such as digital humans and humanoid robot control. However, many current motion generation approach…
HERM: Benchmarking and Enhancing Multimodal LLMs for Human-Centric Understanding
Keliang Li, Zaifei Yang, Jiahe Zhao +5
The significant advancements in visual understanding and instruction following from Multimodal Large Language Models (MLLMs) have opened up more possibilities for broader applicati…
Generalized Semi-Supervised Learning via Self-Supervised Feature Adaptation
Jiachen Liang, Ruibing Hou, Hong Chang +3
Traditional semi-supervised learning (SSL) assumes that the feature distributions of labeled and unlabeled data are consistent which rarely holds in realistic scenarios. In this pa…
MGPT: An Advanced Multimodal, Multitask Framework for Motion Comprehension and Generation
Mingshuang Luo, Ruibing Hou, Zhuo Li +4
This paper presents MGPT, an advanced ultimodal, ultitask framework for otion comprehension and generation. MGPT operates on three funda…
Clothes-Changing Person Re-Identification with Feasibility-Aware Intermediary Matching
Jiahe Zhao, Ruibing Hou, Hong Chang +4
Current clothes-changing person re-identification (re-id) approaches usually perform retrieval based on clothes-irrelevant features, while neglecting the potential of clothes-relev…