28 citations · 30 across the 8 of their papers we have counts for
8 papers
Decoupled Vision-Language System for Multimodal Understanding and Generation
Yifan Xu, Baochen Xiong, Xiaoshan Yang +3
We introduce a new architecture design for multimodal large language models (MLLMs), Libra, capable of both multimodal understanding and generation. Libra architecture contains one…
A Step Toward Federated Pretraining of Multimodal Large Language Models
Baochen Xiong, Yifan Xu, Xiaoshan Yang +3
The rapid evolution of Multimodal Large Language Models (MLLMs) is bottlenecked by the saturation of high-quality public data, while vast amounts of diverse multimodal data remain…
Harmony: A Unified Framework for Modality Incremental Learning
Yaguang Song, Xiaoshan Yang, Dongmei Jiang +2
Incremental learning aims to enable models to continuously acquire knowledge from evolving data streams while preserving previously learned capabilities. While current research pre…
Pilot: Building the Federated Multimodal Instruction Tuning Framework
Baochen Xiong, Xiaoshan Yang, Yaguang Song +2
In this paper, we explore a novel federated multimodal instruction tuning task(FedMIT), which is significant for collaboratively fine-tuning MLLMs on different types of multimodal…
Do We Need to Design Specific Diffusion Models for Different Tasks? Try ONE-PIC
Ming Tao, Bing-Kun Bao, Yaowei Wang +1
Large pretrained diffusion models have demonstrated impressive generation capabilities and have been adapted to various downstream tasks. However, unlike Large Language Models (LLM…
OneRef: Unified One-tower Expression Grounding and Segmentation with Mask Referring Modeling
Linhui Xiao, Xiaoshan Yang, Fang Peng +2
Constrained by the separate encoding of vision and language, existing grounding and referring segmentation works heavily rely on bulky Transformer-based fusion en-/decoders and a v…
StoryImager: A Unified and Efficient Framework for Coherent Story Visualization and Completion
Ming Tao, Bing-Kun Bao, Hao Tang +2
Story visualization aims to generate a series of realistic and coherent images based on a storyline. Current models adopt a frame-by-frame architecture by transforming the pre-trai…
HiVG: Hierarchical Multimodal Fine-grained Modulation for Visual Grounding
Linhui Xiao, Xiaoshan Yang, Fang Peng +2
Visual grounding, which aims to ground a visual region via natural language, is a task that heavily relies on cross-modal alignment. Existing works utilized uni-modal pre-trained m…