479 citations · 1.9k across the 94 of their papers we have counts for
22 papers · 1 filter
TouchStone: Evaluating Vision-Language Models by Language Models
Shuai Bai, Shusheng Yang, Jinze Bai +6
Large vision-language models (LVLMs) have recently witnessed rapid advancements, exhibiting a remarkable capacity for perceiving, understanding, and processing visual information b…
SLPT: Selective Labeling Meets Prompt Tuning on Label-Limited Lesion Segmentation
Fan Bai, Ke Yan, Xiaoyu Bai +6
Medical image analysis using deep learning is often challenged by limited labeled data and high annotation costs. Fine-tuning the entire network in label-limited scenarios can lead…
Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond
Jinze Bai, Shuai Bai, Shusheng Yang +6
In this work, we introduce the Qwen-VL series, a set of large-scale vision-language models (LVLMs) designed to perceive and understand both texts and images. Starting from the Qwen…
Matching in the Wild: Learning Anatomical Embeddings for Multi-Modality Images
Xiaoyu Bai, Fan Bai, Xiaofei Huo +10
Radiotherapists require accurate registration of MR/CT images to effectively use information from both modalities. In a typical registration pipeline, rigid or affine transformatio…
SAMConvex: Fast Discrete Optimization for CT Registration using Self-supervised Anatomical Embedding and Correlation Pyramid
Zi Li, Lin Tian, Tony C. W. Mok +8
Estimating displacement vector field via a cost volume computed in the feature space has shown great success in image registration, but it suffers excessive computation burdens. Mo…
VideoComposer: Compositional Video Synthesis with Motion Controllability
Xiang Wang, Hangjie Yuan, Shiwei Zhang +6
The pursuit of controllability as a higher standard of visual content creation has yielded remarkable progress in customizable image synthesis. However, achieving controllable vide…
Lipschitz Singularities in Diffusion Models
Zhantao Yang, Ruili Feng, Han Zhang +8
Diffusion models, which employ stochastic differential equations to sample images through integrals, have emerged as a dominant class of generative models. However, the rationality…
Cones 2: Customizable Image Synthesis with Multiple Subjects
Zhiheng Liu, Yifei Zhang, Yujun Shen +7
Synthesizing images with user-specified subjects has received growing attention due to its practical applications. Despite the recent success in single subject customization, exist…
ONE-PEACE: Exploring One General Representation Model Toward Unlimited Modalities
Peng Wang, Shijie Wang, Junyang Lin +5
In this work, we explore a scalable way for building a general representation model toward unlimited modalities. We release ONE-PEACE, a highly extensible model with 4B parameters…
ViM: Vision Middleware for Unified Downstream Transferring
Yutong Feng, Biao Gong, Jianwen Jiang +4
Foundation models are pre-trained on massive data and transferred to downstream tasks via fine-tuning. This work presents Vision Middleware (ViM), a new learning paradigm that targ…
Cones: Concept Neurons in Diffusion Models for Customized Generation
Zhiheng Liu, Ruili Feng, Kai Zhu +6
Human brains respond to semantic features of presented stimuli with different neurons. It is then curious whether modern deep neural networks admit a similar behavior pattern. Spec…
Rethinking Efficient Tuning Methods from a Unified Perspective
Zeyinzi Jiang, Chaojie Mao, Ziyuan Huang +3
Parameter-efficient transfer learning (PETL) based on large-scale pre-trained foundation models has achieved great success in various downstream applications. Existing tuning metho…
Composer: Creative and Controllable Image Synthesis with Composable Conditions
Lianghua Huang, Di Chen, Yu Liu +3
Recent large-scale generative models learned on big data are capable of synthesizing incredible images yet suffer from limited controllability. This work offers a new generation pa…
mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Video
Haiyang Xu, Qinghao Ye, Ming Yan +12
Recent years have witnessed a big convergence of language, vision, and multi-modal pretraining. In this work, we present mPLUG-2, a new unified paradigm with modularized design for…
OFASys: A Multi-Modal Multi-Task Learning System for Building Generalist Models
Jinze Bai, Rui Men, Hao Yang +15
Generalist models, which are capable of performing diverse multi-modal tasks in a task-agnostic way within a single model, have been explored recently. Being, hopefully, an alterna…
Chinese CLIP: Contrastive Vision-Language Pretraining in Chinese
An Yang, Junshu Pan, Junyang Lin +4
The tremendous success of CLIP (Radford et al., 2021) has promoted the research and application of contrastive learning for vision-language pretraining. In this work, we construct…
In-N-Out Generative Learning for Dense Unsupervised Video Segmentation
Xiao Pan, Peike Li, Zongxin Yang +5
In this paper, we focus on unsupervised learning for Video Object Segmentation (VOS) which learns visual correspondence (i.e., the similarity between pixel-level features) from unl…
OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework
Peng Wang, An Yang, Rui Men +7
In this work, we pursue a unified paradigm for multimodal pretraining to break the scaffolds of complex task/modality-specific customization. We propose OFA, a Task-Agnostic and Mo…
Low-Rank Subspaces in GANs
Jiapeng Zhu, Ruili Feng, Yujun Shen +4
The latent space of a Generative Adversarial Network (GAN) has been shown to encode rich semantics within some subspaces. To identify these subspaces, researchers typically analyze…
M6-UFC: Unifying Multi-Modal Controls for Conditional Image Synthesis via Non-Autoregressive Generative Transformers
Zhu Zhang, Jianxin Ma, Chang Zhou +6
Conditional image synthesis aims to create an image according to some multi-modal guidance in the forms of textual descriptions, reference images, and image blocks to preserve, as…
Poet: Product-oriented Video Captioner for E-commerce
Shengyu Zhang, Ziqi Tan, Jin Yu +6
In e-commerce, a growing number of user-generated videos are used for product promotion. How to generate video descriptions that narrate the user-preferred product characteristics…
Comprehensive Information Integration Modeling Framework for Video Titling
Shengyu Zhang, Ziqi Tan, Jin Yu +6
In e-commerce, consumer-generated videos, which in general deliver consumers' individual preferences for the different aspects of certain products, are massive in volume. To recomm…