311 citations · 802 across the 39 of their papers we have counts for
41 papers
ViQ: Text-Aligned Visual Quantized Representations at Any Resolution
Xumin Yu, Zuyan Liu, Zhenyu Yang +5
A unified representation for text and vision is a natural pursuit, as it enables simpler multimodal modeling and more efficient training. However, representing images as discrete s…
SparseMM: Head Sparsity Emerges from Visual Concept Responses in MLLMs
Jiahui Wang, Zuyan Liu, Yongming Rao +1
Multimodal Large Language Models (MLLMs) are commonly derived by extending pre-trained Large Language Models (LLMs) with visual capabilities. In this work, we investigate how MLLMs…
Vision Generalist Model: A Survey
Ziyi Wang, Yongming Rao, Shuofeng Sun +8
Recently, we have witnessed the great success of the generalist model in natural language processing. The generalist model is a general framework trained with massive data and is a…
Oryx MLLM: On-Demand Spatial-Temporal Understanding at Arbitrary Resolution
Zuyan Liu, Yuhao Dong, Ziwei Liu +3
Visual data comes in various forms, ranging from small icons of just a few pixels to long videos spanning hours. Existing multi-modal LLMs usually standardize these diverse visual…
Coarse Correspondences Boost Spatial-Temporal Reasoning in Multimodal Language Model
Benlin Liu, Yuhao Dong, Yiqin Wang +6
Multimodal language models (MLLMs) are increasingly being applied in real-world environments, necessitating their ability to interpret 3D spaces and comprehend temporal dynamics. C…
Efficient Inference of Vision Instruction-Following Models with Elastic Cache
Zuyan Liu, Benlin Liu, Jiahui Wang +5
In the field of instruction-following large vision-language models (LVLMs), the efficient deployment of these models faces challenges, notably due to the high memory demands of the…
X-3D: Explicit 3D Structure Modeling for Point Cloud Recognition
Shuofeng Sun, Yongming Rao, Jiwen Lu +1
Numerous prior studies predominantly emphasize constructing relation vectors for individual neighborhood points and generating dynamic kernels for each vector and embedding these i…
Chain-of-Spot: Interactive Reasoning Improves Large Vision-Language Models
Zuyan Liu, Yuhao Dong, Yongming Rao +2
In the realm of vision-language understanding, the proficiency of models in interpreting and reasoning over visual content has become a cornerstone for numerous applications. Howev…
TCOVIS: Temporally Consistent Online Video Instance Segmentation
Junlong Li, Bingyao Yu, Yongming Rao +2
In recent years, significant progress has been made in video instance segmentation (VIS), with many offline and online methods achieving state-of-the-art performance. While offline…
Take-A-Photo: 3D-to-2D Generative Pre-training of Point Cloud Models
Ziyi Wang, Xumin Yu, Yongming Rao +2
With the overwhelming trend of mask image modeling led by MAE, generative pre-training has shown a remarkable potential to boost the performance of fundamental models in 2D vision.…
Unleashing Text-to-Image Diffusion Models for Visual Perception
Wenliang Zhao, Yongming Rao, Zuyan Liu +3
Diffusion models (DMs) have become the new trend of generative models and have demonstrated a powerful ability of conditional synthesis. Among those, text-to-image diffusion models…
UniPC: A Unified Predictor-Corrector Framework for Fast Sampling of Diffusion Models
Wenliang Zhao, Lujia Bai, Yongming Rao +2
Diffusion probabilistic models (DPMs) have demonstrated a very promising ability in high-resolution image synthesis. However, sampling from a pre-trained DPM is time-consuming due…
AdaPoinTr: Diverse Point Cloud Completion with Adaptive Geometry-Aware Transformers
Xumin Yu, Yongming Rao, Ziyi Wang +2
In this paper, we present a new method that reformulates point cloud completion as a set-to-set translation problem and design a new model, called PoinTr, which adopts a Transforme…
FLAG3D: A 3D Fitness Activity Dataset with Language Instruction
Yansong Tang, Jinpeng Liu, Aoyang Liu +6
With the continuously thriving popularity around the world, fitness activity analytic has become an emerging research topic in computer vision. While a variety of new tasks and alg…
PLOT: Prompt Learning with Optimal Transport for Vision-Language Models
Guangyi Chen, Weiran Yao, Xiangchen Song +3
With the increasing attention to large vision-language models such as CLIP, there has been a significant amount of effort dedicated to building efficient prompts. Unlike convention…
P2P: Tuning Pre-trained Image Models for Point Cloud Analysis with Point-to-Pixel Prompting
Ziyi Wang, Xumin Yu, Yongming Rao +2
Nowadays, pre-training big models on large-scale datasets has become a crucial topic in deep learning. The pre-trained models with high representation ability and transferability a…
HorNet: Efficient High-Order Spatial Interactions with Recursive Gated Convolutions
Yongming Rao, Wenliang Zhao, Yansong Tang +3
Recent progress in vision Transformers exhibits great success in various tasks driven by the new spatial modeling mechanism based on dot-product self-attention. In this paper, we s…
Dynamic Spatial Sparsification for Efficient Vision Transformers and Convolutional Neural Networks
Yongming Rao, Zuyan Liu, Wenliang Zhao +2
In this paper, we present a new approach for model acceleration by exploiting spatial sparsity in visual data. We observe that the final prediction in vision Transformers is only b…
SemAffiNet: Semantic-Affine Transformation for Point Cloud Segmentation
Ziyi Wang, Yongming Rao, Xumin Yu +2
Conventional point cloud semantic segmentation methods usually employ an encoder-decoder architecture, where mid-level features are locally aggregated to extract geometric informat…
A Roadmap for Big Model
Sha Yuan, Hanyu Zhao, Shuai Zhao +97
With the rapid development of deep learning, training Big Models (BMs) for multiple downstream tasks becomes a popular paradigm. Researchers have achieved various outcomes in the c…
FineDiving: A Fine-grained Dataset for Procedure-aware Action Quality Assessment
Jinglin Xu, Yongming Rao, Xumin Yu +3
Most existing action quality assessment methods rely on the deep features of an entire video to predict the score, which is less reliable due to the non-transparent inference proce…
SurroundDepth: Entangling Surrounding Views for Self-Supervised Multi-Camera Depth Estimation
Yi Wei, Linqing Zhao, Wenzhao Zheng +5
Depth estimation from images serves as the fundamental step of 3D perception for autonomous driving and is an economical alternative to expensive depth sensors like LiDAR. The temp…
Back to Reality: Weakly-supervised 3D Object Detection with Shape-guided Label Enhancement
Xiuwei Xu, Yifan Wang, Yu Zheng +3
In this paper, we propose a weakly-supervised approach for 3D object detection, which makes it possible to train a strong 3D detector with position-level annotations (i.e. annotati…
Stochastic Trajectory Prediction via Motion Indeterminacy Diffusion
Tianpei Gu, Guangyi Chen, Junlong Li +4
Human behavior has the nature of indeterminacy, which requires the pedestrian trajectory prediction system to model the multi-modality of future motion states. Unlike existing stoc…