115 citations · 573 across the 44 of their papers we have counts for
38 papers · 1 filter
MagicSeg: Open-World Segmentation Pretraining via Counterfactural Diffusion-Based Auto-Generation
Kaixin Cai, Pengzhen Ren, Jianhua Han +4
Open-world semantic segmentation presently relies significantly on extensive image-text pair datasets, which often suffer from a lack of fine-grained pixel annotations on sufficien…
CAP-Net: A Unified Network for 6D Pose and Size Estimation of Categorical Articulated Parts from a Single RGB-D Image
Jingshun Huang, Haitao Lin, Tianyu Wang +3
This paper tackles category-level pose estimation of articulated objects in robotic manipulation tasks and introduces a new benchmark dataset. While recent methods estimate part po…
VidCRAFT3: Camera, Object, and Lighting Control for Image-to-Video Generation
Sixiao Zheng, Zimian Peng, Yanpeng Zhou +4
Controllable image-to-video (I2V) generation transforms a reference image into a coherent video guided by user-specified control signals. While precise control over camera motion,…
VidMan: Exploiting Implicit Dynamics from Video Diffusion Model for Effective Robot Manipulation
Youpeng Wen, Junfan Lin, Yi Zhu +4
Recent advancements utilizing large-scale video data for learning video generation models demonstrate significant potential in understanding complex physical dynamics. It suggests…
UNIT: Unifying Image and Text Recognition in One Vision Encoder
Yi Zhu, Yanpeng Zhou, Chunwei Wang +4
Currently, vision encoder models like Vision Transformers (ViTs) typically excel at image recognition tasks but cannot simultaneously support text recognition like human visual rec…
EMOVA: Empowering Language Models to See, Hear and Speak with Vivid Emotions
Kai Chen, Yunhao Gou, Runhui Huang +28
GPT-4o, an omni-modal model that enables vocal conversations with diverse emotions and tones, marks a milestone for omni-modal foundation models. However, empowering Large Language…
Correctable Landmark Discovery via Large Models for Vision-Language Navigation
Bingqian Lin, Yunshuang Nie, Ziming Wei +5
Vision-Language Navigation (VLN) requires the agent to follow language instructions to reach a target position. A key factor for successful navigation is to align the landmarks imp…
Language-Driven Visual Consensus for Zero-Shot Semantic Segmentation
Zicheng Zhang, Tong Zhang, Yi Zhu +4
The pre-trained vision-language model, exemplified by CLIP, advances zero-shot semantic segmentation by aligning visual features with class embeddings through a transformer decoder…
Towards Deviation-Robust Agent Navigation via Perturbation-Aware Contrastive Learning
Bingqian Lin, Yanxin Long, Yi Zhu +4
Vision-and-language navigation (VLN) asks an agent to follow a given language instruction to navigate through a real 3D environment. Despite significant advances, conventional VLN…
MixReorg: Cross-Modal Mixed Patch Reorganization is a Good Mask Learner for Open-World Semantic Segmentation
Kaixin Cai, Pengzhen Ren, Yi Zhu +5
Recently, semantic segmentation models trained with image-level text supervision have shown promising results in challenging open-world scenarios. However, these models still face…
Prompt Pre-Training with Twenty-Thousand Classes for Open-Vocabulary Visual Recognition
Shuhuai Ren, Aston Zhang, Yi Zhu +5
This work proposes POMP, a prompt pre-training method for vision-language models. Being memory and computation efficient, POMP enables the learned prompt to condense semantic infor…
AIM: Adapting Image Models for Efficient Video Action Recognition
Taojiannan Yang, Yi Zhu, Yusheng Xie +3
Recent vision transformer based video models mostly follow the ``image pre-training then finetuning" paradigm and have achieved great success on multiple video benchmarks. However,…
Actional Atomic-Concept Learning for Demystifying Vision-Language Navigation
Bingqian Lin, Yi Zhu, Xiaodan Liang +2
Vision-Language Navigation (VLN) is a challenging task which requires an agent to align complex visual observations to language instructions to reach the goal position. Most existi…
Towards Geospatial Foundation Models via Continual Pretraining
Matias Mendieta, Boran Han, Xingjian Shi +2
Geospatial technologies are becoming increasingly essential in our world for a wide range of applications, including agriculture, urban planning, and disaster response. To help imp…
ViewCo: Discovering Text-Supervised Segmentation Masks via Multi-View Semantic Consistency
Pengzhen Ren, Changlin Li, Hang Xu +5
Recently, great success has been made in learning visual representations from text supervision, facilitating the emergence of text-supervised semantic segmentation. However, existi…
What Makes for Good Tokenizers in Vision Transformer?
Shengju Qian, Yi Zhu, Wenbo Li +2
The architecture of transformers, which recently witness booming applications in vision tasks, has pivoted against the widespread convolutional paradigm. Relying on the tokenizatio…
CoupAlign: Coupling Word-Pixel with Sentence-Mask Alignments for Referring Image Segmentation
Zicheng Zhang, Yi Zhu, Jianzhuang Liu +2
Referring image segmentation aims at localizing all pixels of the visual objects described by a natural language sentence. Previous works learn to straightforwardly align the sente…
Benchmarking Robustness of Multimodal Image-Text Models under Distribution Shift
Jielin Qiu, Yi Zhu, Xingjian Shi +5
Multimodal image-text models have shown remarkable performance in the past few years. However, evaluating robustness against distribution shifts is crucial before adopting them in…
Visual Prompt Tuning for Test-time Domain Adaptation
Yunhe Gao, Xingjian Shi, Yi Zhu +5
Models should be able to adapt to unseen data during test-time to avoid performance drops caused by inevitable distribution shifts in real-world deployment scenarios. In this work,…
Fine-grained Visual-Text Prompt-Driven Self-Training for Open-Vocabulary Object Detection
Yanxin Long, Jianhua Han, Runhui Huang +4
Inspired by the success of vision-language methods (VLMs) in zero-shot classification, recent works attempt to extend this line of work into object detection by leveraging the loca…
Partial and Asymmetric Contrastive Learning for Out-of-Distribution Detection in Long-Tailed Recognition
Haotao Wang, Aston Zhang, Yi Zhu +4
Existing out-of-distribution (OOD) detection methods are typically benchmarked on training sets with balanced class distributions. However, in real-world applications, it is common…
MixGen: A New Multi-Modal Data Augmentation
Xiaoshuai Hao, Yi Zhu, Srikar Appalaraju +4
Data augmentation is a necessity to enhance data efficiency in deep learning. For vision-language pre-training, data is only augmented either for images or for text in previous wor…
ADAPT: Vision-Language Navigation with Modality-Aligned Action Prompts
Bingqian Lin, Yi Zhu, Zicong Chen +3
Vision-Language Navigation (VLN) is a challenging task that requires an embodied agent to perform action-level modality alignment, i.e., make instruction-asked actions sequentially…
ImpDet: Exploring Implicit Fields for 3D Object Detection
Xuelin Qian, Li Wang, Yi Zhu +3
Conventional 3D object detection approaches concentrate on bounding boxes representation learning with several parameters, i.e., localization, dimension, and orientation. Despite i…