37 citations · 128 across the 32 of their papers we have counts for
22 papers · 1 filter
Scaling Up Audio-Synchronized Visual Animation: An Efficient Training Paradigm
Lin Zhang, Zefan Cai, Yufan Zhou +10
Recent advances in audio-synchronized visual animation enable control of video content using audios from specific classes. However, existing methods rely heavily on expensive manua…
Multimodal LLMs as Customized Reward Models for Text-to-Image Generation
Shijie Zhou, Ruiyi Zhang, Huaisheng Zhu +5
We introduce LLaVA-Reward, an efficient reward model designed to automatically evaluate text-to-image (T2I) generations across multiple perspectives, leveraging pretrained multimod…
OmniPart: Part-Aware 3D Generation with Semantic Decoupling and Structural Cohesion
Yunhan Yang, Yufan Zhou, Yuan-Chen Guo +7
The creation of 3D assets with explicit, editable part structures is crucial for advancing interactive applications, yet most generative methods produce only monolithic shapes, lim…
Towards Visual Text Grounding of Multimodal Large Language Model
Ming Li, Ruiyi Zhang, Jian Chen +7
Despite the existing evolution of Multimodal Large Language Models (MLLMs), a non-neglectable limitation remains in their struggle with visual text grounding, especially in text-ri…
FreeBlend: Advancing Concept Blending with Staged Feedback-Driven Interpolation Diffusion
Yufan Zhou, Haoyu Shen, Huan Wang
Concept blending is a promising yet underexplored area in generative models. While recent approaches, such as embedding mixing and latent modification based on structural sketches,…
A High-Quality Text-Rich Image Instruction Tuning Dataset via Hybrid Instruction Generation
Shijie Zhou, Ruiyi Zhang, Yufan Zhou +1
Large multimodal models still struggle with text-rich images because of inadequate training data. Self-Instruct provides an annotation-free way for generating instruction data, but…
SUGAR: Subject-Driven Video Customization in a Zero-Shot Manner
Yufan Zhou, Ruiyi Zhang, Jiuxiang Gu +3
We present SUGAR, a zero-shot method for subject-driven video customization. Given an input image, SUGAR is capable of generating videos for the subject contained in the image and…
TTVD: Towards a Geometric Framework for Test-Time Adaptation Based on Voronoi Diagram
Mingxi Lei, Chunwei Ma, Meng Ding +3
Deep learning models often struggle with generalization when deploying on real-world data, due to the common distributional shift to the training data. Test-time adaptation (TTA) i…
SV-RAG: LoRA-Contextualizing Adaptation of MLLMs for Long Document Understanding
Jian Chen, Ruiyi Zhang, Yufan Zhou +6
Multimodal large language models (MLLMs) have recently shown great progress in text-rich image understanding, yet they still struggle with complex, multi-page visually-rich documen…
MMR: Evaluating Reading Ability of Large Multimodal Models
Jian Chen, Ruiyi Zhang, Yufan Zhou +3
Large multimodal models (LMMs) have demonstrated impressive capabilities in understanding various types of image, including text-rich images. Most existing text-rich image benchmar…
LLaVA-Read: Enhancing Reading Ability of Multimodal Language Models
Ruiyi Zhang, Yufan Zhou, Jian Chen +3
Large multimodal language models have demonstrated impressive capabilities in understanding and manipulating images. However, many of these models struggle with comprehending inten…
TRINS: Towards Multimodal Language Models that Can Read
Ruiyi Zhang, Yanzhe Zhang, Jian Chen +4
Large multimodal language models have shown remarkable proficiency in understanding and editing images. However, a majority of these visually-tuned models struggle to comprehend th…
ARTIST: Improving the Generation of Text-rich Images with Disentangled Diffusion Models and Large Language Models
Jianyi Zhang, Yufan Zhou, Jiuxiang Gu +5
Diffusion models have demonstrated exceptional capabilities in generating a broad spectrum of visual content, yet their proficiency in rendering text is still limited: they often g…
Toffee: Efficient Million-Scale Dataset Construction for Subject-Driven Text-to-Image Generation
Yufan Zhou, Ruiyi Zhang, Kaizhi Zheng +5
In subject-driven text-to-image generation, recent works have achieved superior performance by training the model on synthetic datasets containing numerous image pairs. Trained on…
Towards Aligned Layout Generation via Diffusion Model with Aesthetic Constraints
Jian Chen, Ruiyi Zhang, Yufan Zhou +4
Controllable layout generation refers to the process of creating a plausible visual arrangement of elements within a graphic design (e.g., document and web designs) with constraint…
Customization Assistant for Text-to-image Generation
Yufan Zhou, Ruiyi Zhang, Jiuxiang Gu +1
Customizing pre-trained text-to-image generation model has attracted massive research interest recently, due to its huge potential in real-world applications. Although existing met…
LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understanding
Yanzhe Zhang, Ruiyi Zhang, Jiuxiang Gu +4
Instruction tuning unlocks the superior capability of Large Language Models (LLM) to interact with humans. Furthermore, recent instruction-following datasets include images as visu…
Enhancing Detail Preservation for Customized Text-to-Image Generation: A Regularization-Free Approach
Yufan Zhou, Ruiyi Zhang, Tong Sun +1
Recent text-to-image generation models have demonstrated impressive capability of generating text-aligned images with high fidelity. However, generating images of novel concept pro…
Shifted Diffusion for Text-to-image Generation
Yufan Zhou, Bingchen Liu, Yizhe Zhu +3
We present Corgi, a novel method for text-to-image generation. Corgi is based on our proposed shifted diffusion model, which achieves better image embedding generation from input t…
Lafite2: Few-shot Text-to-Image Generation
Yufan Zhou, Chunyuan Li, Changyou Chen +2
Text-to-image generation models have progressed considerably in recent years, which can now generate impressive realistic images from arbitrary text. Most of such models are traine…
LAFITE: Towards Language-Free Training for Text-to-Image Generation
Yufan Zhou, Ruiyi Zhang, Changyou Chen +6
One of the major challenges in training text-to-image generation models is the need of a large number of high-quality image-text pairs. While image samples are often easily accessi…
Learning Diverse Stochastic Human-Action Generators by Learning Smooth Latent Transitions
Zhenyi Wang, Ping Yu, Yang Zhao +4
Human-motion generation is a long-standing challenging task due to the requirement of accurately modeling complex and diverse dynamic patterns. Most existing methods adopt sequence…