Publications (76)
WorldSimBench: Towards Video Generation Models as World Simulators
Yiran Qin, Zhelun Shi, Jiwen Yu +10
Recent advancements in predictive models have demonstrated exceptional capabilities in predicting the future state of objects and scenes. However, the lack of categorization based…
Diffusion Model is Secretly a Training-free Open Vocabulary Semantic Segmenter
Jinglong Wang, Xiawei Li, Jing Zhang +5
The pre-trained text-image discriminative models, such as CLIP, has been explored for open-vocabulary semantic segmentation with unsatisfactory results due to the loss of crucial l…
Personalize Anything for Free with Diffusion Transformer
Haoran Feng, Zehuan Huang, Lin Li +2
Personalized image generation aims to produce images of user-specified concepts while enabling flexible editing. Recent training-free approaches, while exhibit higher computational…
ForgeryNet: A Versatile Benchmark for Comprehensive Forgery Analysis
Yinan He, Bei Gan, Siyu Chen +6
The rapid progress of photorealistic synthesis techniques has reached at a critical point where the boundary between real and manipulated images starts to blur. Thus, benchmarking…
MP5: A Multi-modal Open-ended Embodied System in Minecraft via Active Perception
Yiran Qin, Enshen Zhou, Qichang Liu +5
It is a long-lasting goal to design an embodied system that can solve long-horizon open-world tasks in human-like ways. However, existing approaches usually struggle with compound…
VL-SAT: Visual-Linguistic Semantics Assisted Training for 3D Semantic Scene Graph Prediction in Point Cloud
Ziqin Wang, Bowen Cheng, Lichen Zhao +3
The task of 3D semantic scene graph (3DSSG) prediction in the point cloud is challenging since (1) the 3D point cloud only captures geometric structures with limited semantics comp…
HomeGuard: VLM-based Embodied Safeguard for Identifying Contextual Risk in Household Task
Xiaoya Lu, Yijin Zhou, Zeren Chen +6
Vision-Language Models (VLMs) empower embodied agents to execute complex instructions, yet they remain vulnerable to contextual safety risks where benign commands become hazardous…
PV-NAS: Practical Neural Architecture Search for Video Recognition
Zihao Wang, Chen Lin, Lu Sheng +2
Recently, deep learning has been utilized to solve video recognition problem due to its prominent representation ability. Deep neural networks for video tasks is highly customized…
Improving RGB-D Point Cloud Registration by Learning Multi-scale Local Linear Transformation
Ziming Wang, Xiaoliang Huo, Zhenghao Chen +3
Point cloud registration aims at estimating the geometric transformation between two point cloud scans, in which point-wise correspondence estimation is the key to its success. In…
Towards Explainable 3D Grounded Visual Question Answering: A New Benchmark and Strong Baseline
Lichen Zhao, Daigang Cai, Jing Zhang +6
Recently, 3D vision-and-language tasks have attracted increasing research interest. Compared to other vision-and-language tasks, the 3D visual question answering (VQA) task is less…
Repurposing 3D Generative Model for Autoregressive Layout Generation
Haoran Feng, Yifan Niu, Zehuan Huang +4
We introduce LaviGen, a framework that repurposes 3D generative models for 3D layout generation. Unlike previous methods that infer object layouts from textual descriptions, LaviGe…
Unsupervised Bi-directional Flow-based Video Generation from one Snapshot
Lu Sheng, Junting Pan, Jiaming Guo +3
Imagining multiple consecutive frames given one single snapshot is challenging, since it is difficult to simultaneously predict diverse motions from a single image and faithfully g…
From GPT-4 to Gemini and Beyond: Assessing the Landscape of MLLMs on Generalizability, Trustworthiness and Causality through Four Modalities
Chaochao Lu, Chen Qian, Guodong Zheng +33
Multi-modal Large Language Models (MLLMs) have shown impressive abilities in generating reasonable responses with respect to multi-modal contents. However, there is still a wide ga…
Improving Pedestrian Attribute Recognition With Weakly-Supervised Multi-Scale Attribute-Specific Localization
Chufeng Tang, Lu Sheng, Zhaoxiang Zhang +1
Pedestrian attribute recognition has been an emerging research topic in the area of video surveillance. To predict the existence of a particular attribute, it is demanded to locali…
Multi-Agent Amodal Completion: Direct Synthesis with Fine-Grained Semantic Guidance
Hongxing Fan, Lipeng Wang, Haohua Chen +3
Amodal completion, generating invisible parts of occluded objects, is vital for applications like image editing and AR. Prior methods face challenges with data needs, generalizatio…
Stable Diffusion Reference Only: Image Prompt and Blueprint Jointly Guided Multi-Condition Diffusion Model for Secondary Painting
Hao Ai, Lu Sheng
Stable Diffusion and ControlNet have achieved excellent results in the field of image generation and synthesis. However, due to the granularity and method of its control, the effic…
Ouroboros3D: Image-to-3D Generation via 3D-aware Recursive Diffusion
Hao Wen, Zehuan Huang, Yaohui Wang +2
Existing single image-to-3D creation methods typically involve a two-stage process, first generating multi-view images, and then using these images for 3D reconstruction. However,…
Octavius: Mitigating Task Interference in MLLMs via LoRA-MoE
Zeren Chen, Ziqin Wang, Zhen Wang +7
Recent studies have demonstrated Large Language Models (LLMs) can extend their zero-shot generalization capabilities to multimodal learning through instruction tuning. As more moda…
AnimaX: Animating the Inanimate in 3D with Joint Video-Pose Diffusion Models
Zehuan Huang, Haoran Feng, Yangtian Sun +3
We present AnimaX, a feed-forward 3D animation framework that bridges the motion priors of video diffusion models with the controllable structure of skeleton-based animation. Tradi…
Bamboo: Building Mega-Scale Vision Dataset Continually with Human-Machine Synergy
Yuanhan Zhang, Qinghong Sun, Yichun Zhou +7
Large-scale datasets play a vital role in computer vision. But current datasets are annotated blindly without differentiation to samples, making the data collection inefficient and…
MV-Adapter: Multi-view Consistent Image Generation Made Easy
Zehuan Huang, Yuan-Chen Guo, Haoran Wang +4
Existing multi-view image generation methods often make invasive modifications to pre-trained text-to-image (T2I) models and require full fine-tuning, leading to (1) high computati…
EpiDiff: Enhancing Multi-View Synthesis via Localized Epipolar-Constrained Diffusion
Zehuan Huang, Hao Wen, Junting Dong +8
Generating multiview images from a single view facilitates the rapid generation of a 3D mesh conditioned on a single image. Recent methods that introduce 3D global representation i…
ForgeryNet -- Face Forgery Analysis Challenge 2021: Methods and Results
Yinan He, Lu Sheng, Jing Shao +19
The rapid progress of photorealistic synthesis techniques has reached a critical point where the boundary between real and manipulated images starts to blur. Recently, a mega-scale…
Back-tracing Representative Points for Voting-based 3D Object Detection in Point Clouds
Bowen Cheng, Lu Sheng, Shaoshuai Shi +2
3D object detection in point clouds is a challenging vision task that benefits various applications for understanding the 3D visual world. Lots of recent research focuses on how to…
Avatar-Net: Multi-scale Zero-shot Style Transfer by Feature Decoration
Lu Sheng, Ziyi Lin, Jing Shao +1
Zero-shot artistic style transfer is an important image synthesis problem aiming at transferring arbitrary style into content images. However, the trade-off between the generalizat…
HydraPlus-Net: Attentive Deep Features for Pedestrian Analysis
Xihui Liu, Haiyu Zhao, Maoqing Tian +5
Pedestrian analysis plays a vital role in intelligent video surveillance and is a key component for security-centric computer vision systems. Despite that the convolutional neural…
Code-as-Monitor: Constraint-aware Visual Programming for Reactive and Proactive Robotic Failure Detection
Enshen Zhou, Qi Su, Cheng Chi +5
Automatic detection and prevention of open-set failures are crucial in closed-loop robotic systems. Recent studies often struggle to simultaneously identify unexpected failures rea…
Effective LLM Code Refinement via Property-Oriented and Structurally Minimal Feedback
Lehan He, Zeren Chen, Zhe Zhang +2
LLMs excel at code generation, yet ensuring the functional correctness of their outputs remains a persistent challenge. While recent studies have applied Test-Driven Development (T…
Exploring Disentangled Feature Representation Beyond Face Identification
Yu Liu, Fangyin Wei, Jing Shao +3
This paper proposes learning disentangled but complementary face features with minimal supervision by face identification. Specifically, we construct an identity Distilling and Dis…
T2ISafety: Benchmark for Assessing Fairness, Toxicity, and Privacy in Image Generation
Lijun Li, Zhelun Shi, Xuhao Hu +5
Text-to-image (T2I) models have rapidly advanced, enabling the generation of high-quality images from text prompts across various domains. However, these models present notable saf…
Self-Supervised Monocular Depth Estimation in the Dark: Towards Data Distribution Compensation
Haolin Yang, Chaoqiang Zhao, Lu Sheng +1
Nighttime self-supervised monocular depth estimation has received increasing attention in recent years. However, using night images for self-supervision is unreliable because the p…
Assessment of Multimodal Large Language Models in Alignment with Human Values
Zhelun Shi, Zhipin Wang, Hongxing Fan +7
Large Language Models (LLMs) aim to serve as versatile assistants aligned with human values, as defined by the principles of being helpful, honest, and harmless (hhh). However, in…
Optical Flow Guided Feature: A Fast and Robust Motion Representation for Video Action Recognition
Shuyang Sun, Zhanghui Kuang, Wanli Ouyang +2
Motion representation plays a vital role in human action recognition in videos. In this study, we introduce a novel compact motion representation for video action recognition, name…
RH20T-P: A Primitive-Level Robotic Dataset Towards Composable Generalization Agents
Zeren Chen, Zhelun Shi, Xiaoya Lu +8
Achieving generalizability in solving out-of-distribution tasks is one of the ultimate goals of learning robotic manipulation. Recent progress of Vision-Language Models (VLMs) has…
Distortion-aware Transformer in 360° Salient Object Detection
Yinjie Zhao, Lichen Zhao, Qian Yu +3
With the emergence of VR and AR, 360° data attracts increasing attention from the computer vision and multimedia communities. Typically, 360° data is projected into 2D ERP (equir…
VoteHMR: Occlusion-Aware Voting Network for Robust 3D Human Mesh Recovery from Partial Point Clouds
Guanze Liu, Yu Rong, Lu Sheng
3D human mesh recovery from point clouds is essential for various tasks, including AR/VR and human behavior understanding. Previous works in this field either require high-quality…
RoboRefer: Towards Spatial Referring with Reasoning in Vision-Language Models for Robotics
Enshen Zhou, Jingkun An, Cheng Chi +8
Spatial referring is a fundamental capability of embodied robots to interact with the 3D physical world. However, even with the powerful pretrained vision language models (VLMs), r…
MineDreamer: Learning to Follow Instructions via Chain-of-Imagination for Simulated-World Control
Enshen Zhou, Yiran Qin, Zhenfei Yin +5
It is a long-lasting goal to design a generalist-embodied agent that can follow diverse instructions in human-like ways. However, existing approaches often fail to steadily follow…
Multi-modality Affinity Inference for Weakly Supervised 3D Semantic Segmentation
Xiawei Li, Qingyuan Xu, Jing Zhang +4
3D point cloud semantic segmentation has a wide range of applications. Recently, weakly supervised point cloud segmentation methods have been proposed, aiming to alleviate the expe…
TIGeR: Tool-Integrated Geometric Reasoning in Vision-Language Models for Robotics
Yi Han, Enshen Zhou, Shanyu Rong +6
Vision-Language Models (VLMs) have shown remarkable capabilities in spatial reasoning, yet they remain fundamentally limited to qualitative precision and lack the computational pre…
Siamese DETR
Zeren Chen, Gengshi Huang, Wei Li +5
Recent self-supervised methods are mainly designed for representation learning with the base model, e.g., ResNets or ViTs. They cannot be easily transferred to DETR, with task-spec…
Visibility Constrained Generative Model for Depth-based 3D Facial Pose Tracking
Lu Sheng, Jianfei Cai, Tat-Jen Cham +2
In this paper, we propose a generative framework that unifies depth-based 3D facial pose tracking and face model adaptation on-the-fly, in the unconstrained scenarios with heavy oc…
Systematic Reward Gap Optimization for Mitigating VLM Hallucinations
Lehan He, Zeren Chen, Zhelun Shi +3
The success of Direct Preference Optimization (DPO) in mitigating hallucinations in Vision Language Models (VLMs) critically hinges on the true reward gaps within preference pairs.…
DanceFormer: Music Conditioned 3D Dance Generation with Parametric Motion Transformer
Buyu Li, Yongchi Zhao, Zhelun Shi +1
Generating 3D dances from music is an emerged research task that benefits a lot of applications in vision and graphics. Previous works treat this task as sequence generation, howev…
Context and Attribute Grounded Dense Captioning
Guojun Yin, Lu Sheng, Bin Liu +3
Dense captioning aims at simultaneously localizing semantic regions and describing these regions-of-interest (ROIs) with short phrases or sentences in natural language. Previous st…
Zoom-Net: Mining Deep Feature Interactions for Visual Relationship Recognition
Guojun Yin, Lu Sheng, Bin Liu +4
Recognizing visual relationships <subject-predicate-object> among any pair of localized objects is pivotal for image understanding. Previous studies have shown remarkable progress…
CAMP: Cross-Modal Adaptive Message Passing for Text-Image Retrieval
Zihao Wang, Xihui Liu, Hongsheng Li +4
Text-image cross-modal retrieval is a challenging task in the field of language and vision. Most previous approaches independently embed images and sentences into a joint embedding…
VoxHammer: Training-Free Precise and Coherent 3D Editing in Native 3D Space
Lin Li, Zehuan Huang, Haoran Feng +4
3D local editing of specified regions is crucial for game industry and robot interaction. Recent methods typically edit rendered multi-view images and then reconstruct 3D models, b…
HexPlane Representation for 3D Semantic Scene Understanding
Zeren Chen, Yuenan Hou, Yulin Chen +3
In this paper, we introduce the HexPlane representation for 3D semantic scene understanding. Specifically, we first design the View Projection Module (VPM) to project the 3D point…
Towards Spatial Trace with Reasoning in Vision-Language Models for Robotics
Enshen Zhou, Yibo Li, Jingkun An +12
Spatial tracing, as a fundamental embodied interaction ability for robots, is inherently challenging as it requires multi-step metric-grounded reasoning compounded with complex spa…
ChEF: A Comprehensive Evaluation Framework for Standardized Assessment of Multimodal Large Language Models
Zhelun Shi, Zhipin Wang, Hongxing Fan +4
Multimodal Large Language Models (MLLMs) have shown impressive abilities in interacting with visual content with myriad potential downstream tasks. However, even though a list of b…
SegviGen: Repurposing 3D Generative Model for Part Segmentation
Lin Li, Haoran Feng, Zehuan Huang +8
We introduce SegviGen, a framework that repurposes native 3D generative models for 3D part segmentation. Existing pipelines either lift strong 2D priors into 3D via distillation or…
X-Learner: Learning Cross Sources and Tasks for Universal Visual Representation
Yinan He, Gengshi Huang, Siyu Chen +7
In computer vision, pre-training models based on largescale supervised learning have been proven effective over the past few years. However, existing works mostly focus on learning…
Semantics Disentangling for Text-to-Image Generation
Guojun Yin, Bin Liu, Lu Sheng +3
Synthesizing photo-realistic images from text descriptions is a challenging problem. Previous studies have shown remarkable progresses on visual quality of the generated images. In…
Data-Free Generalized Zero-Shot Learning
Bowen Tang, Long Yan, Jing Zhang +3
Deep learning models have the ability to extract rich knowledge from large-scale datasets. However, the sharing of data has become increasingly challenging due to concerns regardin…
Powering One-shot Topological NAS with Stabilized Share-parameter Proxy
Ronghao Guo, Chen Lin, Chuming Li +4
One-shot NAS method has attracted much interest from the research community due to its remarkable training efficiency and capacity to discover high performance models. However, the…
PROMO: Promptable Outfitting for Efficient High-Fidelity Virtual Try-On
Haohua Chen, Tianze Zhou, Wei Zhu +8
Virtual Try-on (VTON) has become a core capability for online retail, where realistic try-on results provide reliable fit guidance, reduce returns, and benefit both consumers and m…
GS3D: An Efficient 3D Object Detection Framework for Autonomous Driving
Buyu Li, Wanli Ouyang, Lu Sheng +2
We present an efficient 3D object detection framework based on a single RGB image in the scenario of autonomous driving. Our efforts are put on extracting the underlying 3D informa…
ELBO-T2IAlign: A Generic ELBO-Based Method for Calibrating Pixel-level Text-Image Alignment in Diffusion Models
Qin Zhou, Zhiyang Zhang, Jinglong Wang +5
Diffusion models excel at image generation. Recent studies have shown that these models not only generate high-quality images but also encode text-image alignment information throu…
Geometrically-Constrained Agent for Spatial Reasoning
Zeren Chen, Xiaoya Lu, Zhijie Zheng +6
Vision Language Models (VLMs) exhibit a fundamental semantic-to-geometric gap in spatial reasoning: they excel at qualitative semantic inference but their reasoning operates within…
From Parts to Whole: A Unified Reference Framework for Controllable Human Image Generation
Zehuan Huang, Hongxing Fan, Lipeng Wang +1
Recent advancements in controllable human image generation have led to zero-shot generation using structural signals (e.g., pose, depth) or facial appearance. Yet, generating human…
Video Generation from Single Semantic Label Map
Junting Pan, Chengyu Wang, Xu Jia +4
This paper proposes the novel task of video generation conditioned on a SINGLE semantic label map, which provides a good balance between flexibility and quality in the generation p…
DeepSeek: Paradigm Shifts and Technical Evolution in Large AI Models
Luolin Xiong, Haofen Wang, Xi Chen +7
DeepSeek, a Chinese Artificial Intelligence (AI) startup, has released their V3 and R1 series models, which attracted global attention due to their low cost, high performance, and…
Multi-Label Image Classification via Knowledge Distillation from Weakly-Supervised Detection
Yongcheng Liu, Lu Sheng, Jing Shao +3
Multi-label image classification is a fundamental but challenging task towards general visual understanding. Existing methods found the region-level cues (e.g., features from RoIs)…
LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmark
Zhenfei Yin, Jiong Wang, Jianjian Cao +9
Large language models have emerged as a promising approach towards achieving general-purpose AI agents. The thriving open-source LLM community has greatly accelerated the developme…
R-Stitch: Dynamic Trajectory Stitching for Efficient Reasoning
Zhuokun Chen, Zeren Chen, Jiahao He +4
Chain-of-thought (CoT) enhances the problem-solving ability of large language models (LLMs) but incurs substantial inference cost due to long autoregressive trajectories. Existing…
InterMoE: Individual-Specific 3D Human Interaction Generation via Dynamic Temporal-Selective MoE
Lipeng Wang, Hongxing Fan, Haohua Chen +2
Generating high-quality human interactions holds significant value for applications like virtual reality and robotics. However, existing methods often fail to preserve unique indiv…
SketchSampler: Sketch-based 3D Reconstruction via View-dependent Depth Sampling
Chenjian Gao, Qian Yu, Lu Sheng +2
Reconstructing a 3D shape based on a single sketch image is challenging due to the large domain gap between a sparse, irregular sketch and a regular, dense 3D shape. Existing works…
Thinking in Frequency: Face Forgery Detection by Mining Frequency-aware Clues
Yuyang Qian, Guojun Yin, Lu Sheng +2
As realistic facial manipulation technologies have achieved remarkable progress, social concerns about potential malicious abuse of these technologies bring out an emerging researc…
Fast-BEV: A Fast and Strong Bird's-Eye View Perception Baseline
Yangguang Li, Bin Huang, Zeren Chen +8
Recently, perception task based on Bird's-Eye View (BEV) representation has drawn more and more attention, and BEV representation is promising as the foundation for next-generation…
Reasoning-Driven Amodal Completion: Collaborative Agents and Perceptual Evaluation
Hongxing Fan, Shuyu Zhao, Jiayang Ao +1
Amodal completion, the task of inferring invisible object parts, faces significant challenges in maintaining semantic consistency and structural integrity. Prior progressive approa…
IS-Bench: Evaluating Interactive Safety of VLM-Driven Embodied Agents in Daily Household Tasks
Xiaoya Lu, Zeren Chen, Xuhao Hu +5
Flawed planning from VLM-driven embodied agents poses significant safety hazards, hindering their deployment in real-world household tasks. However, existing static, non-interactiv…
ProGuard: Towards Proactive Multimodal Safeguard
Shaohan Yu, Lijun Li, Chenyang Si +2
The rapid evolution of generative models has led to a continuous emergence of multimodal safety risks, exposing the limitations of existing defense methods. To address these challe…
Morphing and Sampling Network for Dense Point Cloud Completion
Minghua Liu, Lu Sheng, Sheng Yang +2
3D point cloud completion, the task of inferring the complete geometric shape from a partial point cloud, has been attracting attention in the community. For acquiring high-fidelit…
MIDI: Multi-Instance Diffusion for Single Image to 3D Scene Generation
Zehuan Huang, Yuan-Chen Guo, Xingqiao An +7
This paper introduces MIDI, a novel paradigm for compositional 3D scene generation from a single image. Unlike existing methods that rely on reconstruction or retrieval techniques…
Adaptive Gradient Method with Resilience and Momentum
Jie Liu, Chen Lin, Chuming Li +4
Several variants of stochastic gradient descent (SGD) have been proposed to improve the learning effectiveness and efficiency when training deep neural networks, among which some r…