28 citations · 50 across the 21 of their papers we have counts for
21 papers
A Visual Dependence-Aware Framework for Multimodal Unsupervised Continual Post-Training
Kaichen Li, Zhilin Zhu, Jianhao Huang +7
In this paper, we explore a novel task of Multimodal Unsupervised Continual Post-Training (MU-CPT), enabling deployed MLLMs to continually evolve from streaming unlabeled data. Exi…
UniFed-VLM: Federated Instruction Tuning for Vision-Language Models with Multiple Heterogeneity
Pengyu Wang, Baochen Xiong, Xiaoshan Yang +4
Vision-Language Models (VLMs) have demonstrated strong performance in multimodal understanding and generation. However, fine-tuning of VLMs typically relies on centralized data, wh…
Decoupled Vision-Language System for Multimodal Understanding and Generation
Yifan Xu, Baochen Xiong, Xiaoshan Yang +3
We introduce a new architecture design for multimodal large language models (MLLMs), Libra, capable of both multimodal understanding and generation. Libra architecture contains one…
VTM-Nav: Harnessing Cross-Episode Experience for Object-Goal Navigation with Hierarchical Visual-Topological Memory
Xiaoran Xu, Yupeng Wu, Tianyu Xue +4
Training-free ObjectNav agents increasingly use vision-language models (VLMs), yet typically discard acquired scene knowledge after each request. We study cross-episode ObjectNav,…
DOME: Learning Transferable Domain Variables from Sparse Supervision for Test-Time Adaptation
Xiaoran Xu, Yifan Xu, Yupeng Wu +2
Test-time adaptation (TTA) aims to align a model to shifting test domains using only unlabeled streaming data. Most existing methods implicitly infer a single global domain distrib…
Boosting Multimodal Federated Learning via Chained Modality Optimization
Zixin Zhang, Fan Qi, Shuai Li +2
Multimodal Federated Learning (MMFL) enables privacy-preserving collaborative learning across decentralized clients with heterogeneous data and modality availability. However, most…
General Covariant Action Modeling: Constructing Generalized Manifolds via Spatio-Temporal Decoupling
Huaihai Lyu, Chaofan Chen, Mingyu Cao +2
Achieving robust generalization from limited data is a central challenge in embodied intelligence. Prevailing methods fail by regressing absolute coordinates, which violates the pr…
Replacing Parameters with Preferences: Federated Alignment of Heterogeneous Vision-Language Models
Shule Lu, Yujing Wang, Hainan Zhang +5
Vision-Language Models (VLMs) have broad potential in privacy-sensitive domains such as healthcare and finance, yet strict data-sharing constraints render centralized training infe…
PivotMerge: Bridging Heterogeneous Multimodal Pre-training via Post-Alignment Model Merging
Zibo Shao, Baochen Xiong, Xiaoshan Yang +4
Multimodal Large Language Models (MLLMs) rely on multimodal pre-training over diverse data sources, where different datasets often induce complementary cross-modal alignment capabi…
A Step Toward Federated Pretraining of Multimodal Large Language Models
Baochen Xiong, Yifan Xu, Xiaoshan Yang +3
The rapid evolution of Multimodal Large Language Models (MLLMs) is bottlenecked by the saturation of high-quality public data, while vast amounts of diverse multimodal data remain…
Replacing Parameters with Preferences: Federated Alignment of Heterogeneous Vision-Language Models
Shule Lu, Yujing Wang, Hainan Zhang +5
VLMs have broad potential in privacy-sensitive domains such as healthcare and finance, yet strict data-sharing constraints render centralized training infeasible. FL mitigates this…
Towards Domain-Generalized Open-Vocabulary Object Detection: A Progressive Domain-invariant Cross-modal Alignment Method
Xiaoran Xu, Xiaoshan Yang, Jiangang Yang +3
Open-Vocabulary Object Detection (OVOD) has achieved remarkable success in generalizing to novel categories. However, this success often rests on the implicit assumption of domain…
Modality-Collaborative Low-Rank Decomposers for Few-Shot Video Domain Adaptation
Yuyang Wanyan, Xiaoshan Yang, Weiming Dong +1
In this paper, we study the challenging task of Few-Shot Video Domain Adaptation (FSVDA). The multimodal nature of videos introduces unique challenges, necessitating the simultaneo…
Harmony: A Unified Framework for Modality Incremental Learning
Yaguang Song, Xiaoshan Yang, Dongmei Jiang +2
Incremental learning aims to enable models to continuously acquire knowledge from evolving data streams while preserving previously learned capabilities. While current research pre…
Pilot: Building the Federated Multimodal Instruction Tuning Framework
Baochen Xiong, Xiaoshan Yang, Yaguang Song +2
In this paper, we explore a novel federated multimodal instruction tuning task(FedMIT), which is significant for collaboratively fine-tuning MLLMs on different types of multimodal…
Do We Need to Design Specific Diffusion Models for Different Tasks? Try ONE-PIC
Ming Tao, Bing-Kun Bao, Yaowei Wang +1
Large pretrained diffusion models have demonstrated impressive generation capabilities and have been adapted to various downstream tasks. However, unlike Large Language Models (LLM…
Toward Visual Grounding: A Survey
Linhui Xiao, Xiaoshan Yang, Xiangyuan Lan +2
Visual Grounding, also known as Referring Expression Comprehension and Phrase Grounding, aims to ground the specific region(s) within the image(s) based on the given expression tex…
OneRef: Unified One-tower Expression Grounding and Segmentation with Mask Referring Modeling
Linhui Xiao, Xiaoshan Yang, Fang Peng +2
Constrained by the separate encoding of vision and language, existing grounding and referring segmentation works heavily rely on bulky Transformer-based fusion en-/decoders and a v…
A Comprehensive Review of Few-shot Action Recognition
Yuyang Wanyan, Xiaoshan Yang, Weiming Dong +1
Few-shot action recognition aims to address the high cost and impracticality of manually labeling complex and variable video data in action recognition. It requires accurately clas…
Libra: Building Decoupled Vision System on Large Language Models
Yifan Xu, Xiaoshan Yang, Yaguang Song +1
In this work, we introduce Libra, a prototype model with a decoupled vision system on a large language model (LLM). The decoupled vision system decouples inner-modal modeling and c…
HiVG: Hierarchical Multimodal Fine-grained Modulation for Visual Grounding
Linhui Xiao, Xiaoshan Yang, Fang Peng +2
Visual grounding, which aims to ground a visual region via natural language, is a task that heavily relies on cross-modal alignment. Existing works utilized uni-modal pre-trained m…