8 citations · 27 across the 21 of their papers we have counts for
21 papers
CURV: Enhancing Chart Understanding Through Curriculum Visual Grounded Reasoning
Xuehang Guo, Pingyue Zhang, Ruiyi Zhang +6
Chart question answering (CQA) requires multimodal large language models (MLLMs) to integrate visual comprehension with logical reasoning, yet current models struggle with accurate…
MiLDEdit: Reasoning-Based Multi-Layer Design Document Editing
Zihao Lin, Wanrong Zhu, Jiuxiang Gu +8
Real-world design documents (e.g., posters) are inherently multi-layered, combining decoration, text, and images. Editing them from natural-language instructions requires fine-grai…
OIDA-QA: A Multimodal Benchmark for Analyzing the Opioid Industry Documents Archive
Xuan Shen, Brian Wingenroth, Zichao Wang +12
The opioid crisis represents a significant moment in public health that reveals systemic shortcomings across regulatory systems, healthcare practices, corporate governance, and pub…
Scaling Up Audio-Synchronized Visual Animation: An Efficient Training Paradigm
Lin Zhang, Zefan Cai, Yufan Zhou +10
Recent advances in audio-synchronized visual animation enable control of video content using audios from specific classes. However, existing methods rely heavily on expensive manua…
Towards Visual Text Grounding of Multimodal Large Language Model
Ming Li, Ruiyi Zhang, Jian Chen +7
Despite the existing evolution of Multimodal Large Language Models (MLLMs), a non-neglectable limitation remains in their struggle with visual text grounding, especially in text-ri…
MDocAgent: A Multi-Modal Multi-Agent Framework for Document Understanding
Siwei Han, Peng Xia, Ruiyi Zhang +4
Document Question Answering (DocQA) is a very common task. Existing methods using Large Language Models (LLMs) or Large Vision Language Models (LVLMs) and Retrieval Augmented Gener…
Numerical Pruning for Efficient Autoregressive Models
Xuan Shen, Zhao Song, Yufa Zhou +12
Transformers have emerged as the leading architecture in deep learning, proving to be versatile and highly effective across diverse domains beyond language and image processing. Ho…
SUGAR: Subject-Driven Video Customization in a Zero-Shot Manner
Yufan Zhou, Ruiyi Zhang, Jiuxiang Gu +3
We present SUGAR, a zero-shot method for subject-driven video customization. Given an input image, SUGAR is capable of generating videos for the subject contained in the image and…
TextLap: Customizing Language Models for Text-to-Layout Planning
Jian Chen, Ruiyi Zhang, Yufan Zhou +4
Automatic generation of graphical layouts is crucial for many real-world applications, including designing posters, flyers, advertisements, and graphical user interfaces. Given the…
MMR: Evaluating Reading Ability of Large Multimodal Models
Jian Chen, Ruiyi Zhang, Yufan Zhou +3
Large multimodal models (LMMs) have demonstrated impressive capabilities in understanding various types of image, including text-rich images. Most existing text-rich image benchmar…
LLaVA-Read: Enhancing Reading Ability of Multimodal Language Models
Ruiyi Zhang, Yufan Zhou, Jian Chen +3
Large multimodal language models have demonstrated impressive capabilities in understanding and manipulating images. However, many of these models struggle with comprehending inten…
TRINS: Towards Multimodal Language Models that Can Read
Ruiyi Zhang, Yanzhe Zhang, Jian Chen +4
Large multimodal language models have shown remarkable proficiency in understanding and editing images. However, a majority of these visually-tuned models struggle to comprehend th…
ARTIST: Improving the Generation of Text-rich Images with Disentangled Diffusion Models and Large Language Models
Jianyi Zhang, Yufan Zhou, Jiuxiang Gu +5
Diffusion models have demonstrated exceptional capabilities in generating a broad spectrum of visual content, yet their proficiency in rendering text is still limited: they often g…
Toffee: Efficient Million-Scale Dataset Construction for Subject-Driven Text-to-Image Generation
Yufan Zhou, Ruiyi Zhang, Kaizhi Zheng +5
In subject-driven text-to-image generation, recent works have achieved superior performance by training the model on synthetic datasets containing numerous image pairs. Trained on…
Improve Temporal Awareness of LLMs for Sequential Recommendation
Zhendong Chu, Zichao Wang, Ruiyi Zhang +3
Large language models (LLMs) have demonstrated impressive zero-shot abilities in solving a wide range of general-purpose tasks. However, it is empirically found that LLMs fall shor…
Automatic Layout Planning for Visually-Rich Documents with Instruction-Following Models
Wanrong Zhu, Jennifer Healey, Ruiyi Zhang +2
Recent advancements in instruction-following models have made user interactions with models more user-friendly and efficient, broadening their applicability. In graphic design, non…
SOHES: Self-supervised Open-world Hierarchical Entity Segmentation
Shengcao Cao, Jiuxiang Gu, Jason Kuen +7
Open-world entity segmentation, as an emerging computer vision task, aims at segmenting entities in images without being restricted by pre-defined classes, offering impressive gene…
Customization Assistant for Text-to-image Generation
Yufan Zhou, Ruiyi Zhang, Jiuxiang Gu +1
Customizing pre-trained text-to-image generation model has attracted massive research interest recently, due to its huge potential in real-world applications. Although existing met…
Token-Level Adversarial Prompt Detection Based on Perplexity Measures and Contextual Information
Zhengmian Hu, Gang Wu, Saayan Mitra +4
In recent years, Large Language Models (LLM) have emerged as pivotal tools in various applications. However, these models are susceptible to adversarial prompt attacks, where attac…
AutoDAN: Interpretable Gradient-Based Adversarial Attacks on Large Language Models
Sicheng Zhu, Ruiyi Zhang, Bang An +6
Safety alignment of Large Language Models (LLMs) can be compromised with manual jailbreak attacks and (automatic) adversarial attacks. Recent studies suggest that defending against…
Enhancing Detail Preservation for Customized Text-to-Image Generation: A Regularization-Free Approach
Yufan Zhou, Ruiyi Zhang, Tong Sun +1
Recent text-to-image generation models have demonstrated impressive capability of generating text-aligned images with high fidelity. However, generating images of novel concept pro…