85 citations · 594 across the 81 of their papers we have counts for
75 papers · 1 filter
ViT-Lens: Towards Omni-modal Representations
Weixian Lei, Yixiao Ge, Kun Yi +6
Aiming to advance AI agents, large foundation models significantly improve reasoning and instruction execution, yet the current focus on vision and language neglects the potential…
Exploring Model Transferability through the Lens of Potential Energy
Xiaotong Li, Zixuan Hu, Yixiao Ge +2
Transfer learning has become crucial in computer vision tasks due to the vast availability of pre-trained deep learning models. However, selecting the optimal pre-trained model fro…
Planting a SEED of Vision in Large Language Model
Yuying Ge, Yixiao Ge, Ziyun Zeng +2
We present SEED, an elaborate image tokenizer that empowers Large Language Models (LLMs) with the emergent ability to SEE and Draw at the same time. Research on image tokenizers ha…
DeSRA: Detect and Delete the Artifacts of GAN-based Real-World Super-Resolution Models
Liangbin Xie, Xintao Wang, Xiangyu Chen +4
Image super-resolution (SR) with generative adversarial networks (GAN) has achieved great success in restoring realistic details. However, it is notorious that GAN-based SR models…
DreamDiffusion: Generating High-Quality Images from Brain EEG Signals
Yunpeng Bai, Xintao Wang, Yan-pei Cao +3
This paper introduces DreamDiffusion, a novel method for generating high-quality images directly from brain electroencephalogram (EEG) signals, without the need to translate though…
PTVD: A Large-Scale Plot-Oriented Multimodal Dataset Based on Television Dramas
Chen Li, Xutan Peng, Teng Wang +5
Art forms such as movies and television (TV) dramas are reflections of the real world, which have attracted much attention from the multimodal learning community recently. However,…
TaCA: Upgrading Your Visual Foundation Model with Task-agnostic Compatible Adapter
Binjie Zhang, Yixiao Ge, Xuyuan Xu +2
Visual foundation models like CLIP excel in learning feature representations from extensive datasets through self-supervised methods, demonstrating remarkable transfer learning and…
Sticker820K: Empowering Interactive Retrieval with Stickers
Sijie Zhao, Yixiao Ge, Zhongang Qi +4
Stickers have become a ubiquitous part of modern-day communication, conveying complex emotions through visual imagery. To facilitate the development of more powerful algorithms for…
Make-Your-Video: Customized Video Generation Using Textual and Structural Guidance
Jinbo Xing, Menghan Xia, Yuxin Liu +9
Creating a vivid video from the event or scenario in our imagination is a truly fascinating experience. Recent advancements in text-to-video synthesis have unveiled the potential t…
Inserting Anybody in Diffusion Models via Celeb Basis
Ge Yuan, Xiaodong Cun, Yong Zhang +5
Exquisite demand exists for customizing the pretrained large text-to-image model, , Stable Diffusion, to generate innovative concepts, such as the users themselves.…
GPT4Tools: Teaching Large Language Model to Use Tools via Self-instruction
Rui Yang, Lin Song, Yanwei Li +4
This paper aims to efficiently enable Large Language Models (LLMs) to use multimodal tools. Advanced proprietary LLMs, such as ChatGPT and GPT-4, have shown great potential for too…
TaleCrafter: Interactive Story Visualization with Multiple Characters
Yuan Gong, Youxin Pang, Xiaodong Cun +8
Accurate Story visualization requires several necessary elements, such as identity consistency across frames, the alignment between plain text and visual content, and a reasonable…
TVTSv2: Learning Out-of-the-box Spatiotemporal Visual Representations at Scale
Ziyun Zeng, Yixiao Ge, Zhan Tong +3
The ultimate goal for foundation models is realizing task-agnostic, i.e., supporting out-of-the-box usage without task-specific fine-tuning. Although breakthroughs have been made i…
What Makes for Good Visual Tokenizers for Large Language Models?
Guangzhi Wang, Yixiao Ge, Xiaohan Ding +2
We empirically investigate proper pre-training methods to build good visual tokenizers, making Large Language Models (LLMs) powerful Multimodal Large Language Models (MLLMs). In ou…
-Tuning: Transferring Multimodal Foundation Models with Optimal Multi-task Interpolation
Chengyue Wu, Teng Wang, Yixiao Ge +4
Foundation models have achieved great advances in multi-task learning with a unified interface of unimodal and multimodal tasks. However, the potential of such multi-task learners…
Mix-of-Show: Decentralized Low-Rank Adaptation for Multi-Concept Customization of Diffusion Models
Yuchao Gu, Xintao Wang, Jay Zhangjie Wu +10
Public large-scale text-to-image diffusion models, such as Stable Diffusion, have gained significant attention from the community. These models can be easily customized for new con…
Caption Anything: Interactive Image Description with Diverse Multimodal Controls
Teng Wang, Jinrui Zhang, Junjie Fei +5
Controllable image captioning is an emerging multimodal topic that aims to describe the image with natural language following human purpose, , looking at the specifi…
Improved Test-Time Adaptation for Domain Generalization
Liang Chen, Yong Zhang, Yibing Song +2
The main challenge in domain generalization (DG) is to handle the distribution shift problem that lies between the training and test data. Recent studies suggest that test-time tra…
Follow Your Pose: Pose-Guided Text-to-Video Generation using Pose-Free Videos
Yue Ma, Yingqing He, Xiaodong Cun +5
Generating text-editable and pose-controllable character videos have an imperious demand in creating various digital human. Nevertheless, this task has been restricted by the absen…
DropMAE: Learning Representations via Masked Autoencoders with Spatial-Attention Dropout for Temporal Matching Tasks
Qiangqiang Wu, Tianyu Yang, Ziquan Liu +3
This paper studies masked autoencoder (MAE) video pre-training for various temporal matching-based downstream tasks, i.e., object-level tracking tasks including video object tracki…
Accelerating Vision-Language Pretraining with Free Language Modeling
Teng Wang, Yixiao Ge, Feng Zheng +4
The state of the arts in vision-language pretraining (VLP) achieves exemplary performance but suffers from high training costs resulting from slow convergence and long training tim…
HMC: Hierarchical Mesh Coarsening for Skeleton-free Motion Retargeting
Haoyu Wang, Shaoli Huang, Fang Zhao +2
We present a simple yet effective method for skeleton-free motion retargeting. Previous methods transfer motion between high-resolution meshes, failing to preserve the inherent loc…
T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models
Chong Mou, Xintao Wang, Liangbin Xie +5
The incredible generative ability of large-scale text-to-image (T2I) models has demonstrated strong power of learning complex structures and meaningful semantics. However, relying…
Skinned Motion Retargeting with Residual Perception of Motion Semantics & Geometry
Jiaxu Zhang, Junwu Weng, Di Kang +7
A good motion retargeting cannot be reached without reasonable consideration of source-target differences on both the skeleton and shape geometry levels. In this work, we propose a…