285 citations · 333 across the 13 of their papers we have counts for
13 papers
Bridging Writing Manner Gap in Visual Instruction Tuning by Creating LLM-aligned Instructions
Dong Jing, Nanyi Fei, Zhiwu Lu
In the realm of Large Multi-modal Models (LMMs), the instruction quality during the visual instruction tuning stage significantly influences the performance of modality alignment.…
Leveraging Large Vision-Language Model as User Intent-aware Encoder for Composed Image Retrieval
Zelong Sun, Dong Jing, Guoxing Yang +2
Composed Image Retrieval (CIR) aims to retrieve target images from candidate set using a hybrid-modality query consisting of a reference image and a relative caption that describes…
Awaker2.5-VL: Stably Scaling MLLMs with Parameter-Efficient Mixture of Experts
Jinqiang Long, Yanqi Dai, Guoxing Yang +4
As the research of Multimodal Large Language Models (MLLMs) becomes popular, an advancing MLLM model is typically required to handle various textual and visual tasks (e.g., VQA, De…
Improvable Gap Balancing for Multi-Task Learning
Yanqi Dai, Nanyi Fei, Zhiwu Lu
In multi-task learning (MTL), gradient balancing has recently attracted more research interest than loss balancing since it often leads to better performance. However, loss balanci…
VDT: General-purpose Video Diffusion Transformers via Mask Modeling
Haoyu Lu, Guoxing Yang, Nanyi Fei +4
This work introduces Video Diffusion Transformer (VDT), which pioneers the use of transformers in diffusion-based video generation. It features transformer blocks with modularized…
LGDN: Language-Guided Denoising Network for Video-Language Modeling
Haoyu Lu, Mingyu Ding, Nanyi Fei +2
Video-language modeling has attracted much attention with the rapid growth of web videos. Most existing methods assume that the video frames and text description are semantically c…
Multimodal foundation models are better simulators of the human brain
Haoyu Lu, Qiongyi Zhou, Nanyi Fei +8
Multimodal learning, especially large-scale multimodal pre-training, has developed rapidly over the past few years and led to the greatest advances in artificial intelligence (AI).…
COTS: Collaborative Two-Stream Vision-Language Pre-Training Model for Cross-Modal Retrieval
Haoyu Lu, Nanyi Fei, Yuqi Huo +3
Large-scale single-stream pre-training has shown dramatic performance in image-text retrieval. Regrettably, it faces low inference efficiency due to heavy attention layers. Recentl…
A Roadmap for Big Model
Sha Yuan, Hanyu Zhao, Shuai Zhao +97
With the rapid development of deep learning, training Big Models (BMs) for multiple downstream tasks becomes a popular paradigm. Researchers have achieved various outcomes in the c…
Towards artificial general intelligence via a multimodal foundation model
Nanyi Fei, Zhiwu Lu, Yizhao Gao +9
The fundamental goal of artificial intelligence (AI) is to mimic the core cognitive activities of human. Despite tremendous success in the AI research, most of existing methods hav…
Contrastive Prototype Learning with Augmented Embeddings for Few-Shot Learning
Yizhao Gao, Nanyi Fei, Guangzhen Liu +3
Most recent few-shot learning (FSL) methods are based on meta-learning with episodic training. In each meta-training episode, a discriminative feature embedding and/or classifier a…
Meta-Learning across Meta-Tasks for Few-Shot Learning
Nanyi Fei, Zhiwu Lu, Yizhao Gao +3
Existing meta-learning based few-shot learning (FSL) methods typically adopt an episodic training strategy whereby each episode contains a meta-task. Across episodes, these tasks a…
Zero-Shot Learning with Sparse Attribute Propagation
Nanyi Fei, Jiechao Guan, Zhiwu Lu +2
Zero-shot learning (ZSL) aims to recognize a set of unseen classes without any training images. The standard approach to ZSL requires a set of training images annotated with seen c…