285 citations · 966 across the 40 of their papers we have counts for
40 papers
Improvable Gap Balancing for Multi-Task Learning
Yanqi Dai, Nanyi Fei, Zhiwu Lu
In multi-task learning (MTL), gradient balancing has recently attracted more research interest than loss balancing since it often leads to better performance. However, loss balanci…
VDT: General-purpose Video Diffusion Transformers via Mask Modeling
Haoyu Lu, Guoxing Yang, Nanyi Fei +4
This work introduces Video Diffusion Transformer (VDT), which pioneers the use of transformers in diffusion-based video generation. It features transformer blocks with modularized…
UniAdapter: Unified Parameter-Efficient Transfer Learning for Cross-modal Modeling
Haoyu Lu, Yuqi Huo, Guoxing Yang +4
Large-scale vision-language pre-trained models have shown promising transferability to various downstream tasks. As the size of these foundation models and the number of downstream…
Text2Poster: Laying out Stylized Texts on Retrieved Images
Chuhao Jin, Hongteng Xu, Ruihua Song +1
Poster generation is a significant task for a wide range of applications, which is often time-consuming and requires lots of manual editing and artistic experience. In this paper,…
TikTalk: A Video-Based Dialogue Dataset for Multi-Modal Chitchat in Real World
Hongpeng Lin, Ludan Ruan, Wenke Xia +8
To facilitate the research on intelligent and human-like chatbots with multi-modal context, we introduce a new video-based multi-modal dialogue dataset, called TikTalk. We collect…
A Molecular Multimodal Foundation Model Associating Molecule Graphs with Natural Language
Bing Su, Dazhao Du, Zhao Yang +6
Although artificial intelligence (AI) has made significant progress in understanding molecules in a wide range of fields, existing models generally acquire the single cognitive abi…
LGDN: Language-Guided Denoising Network for Video-Language Modeling
Haoyu Lu, Mingyu Ding, Nanyi Fei +2
Video-language modeling has attracted much attention with the rapid growth of web videos. Most existing methods assume that the video frames and text description are semantically c…
Multimodal foundation models are better simulators of the human brain
Haoyu Lu, Qiongyi Zhou, Nanyi Fei +8
Multimodal learning, especially large-scale multimodal pre-training, has developed rapidly over the past few years and led to the greatest advances in artificial intelligence (AI).…
COTS: Collaborative Two-Stream Vision-Language Pre-Training Model for Cross-Modal Retrieval
Haoyu Lu, Nanyi Fei, Yuqi Huo +3
Large-scale single-stream pre-training has shown dramatic performance in image-text retrieval. Regrettably, it faces low inference efficiency due to heavy attention layers. Recentl…
Towards artificial general intelligence via a multimodal foundation model
Nanyi Fei, Zhiwu Lu, Yizhao Gao +9
The fundamental goal of artificial intelligence (AI) is to mimic the core cognitive activities of human. Despite tremendous success in the AI research, most of existing methods hav…
HR-NAS: Searching Efficient High-Resolution Neural Architectures with Lightweight Transformers
Mingyu Ding, Xiaochen Lian, Linjie Yang +4
High-resolution representations (HR) are essential for dense prediction tasks such as segmentation, detection, and pose estimation. Learning HR representations is typically ignored…
Pre-Trained Models: Past, Present and Future
Xu Han, Zhengyan Zhang, Ning Ding +21
Large-scale pre-trained models (PTMs) such as BERT and GPT have recently achieved great success and become a milestone in the field of artificial intelligence (AI). Owing to sophis…
Learning Versatile Neural Architectures by Propagating Network Codes
Mingyu Ding, Yuqi Huo, Haoyu Lu +5
This work explores how to design a single neural network capable of adapting to multiple heterogeneous vision tasks, such as image segmentation, 3D detection, and video recognition…
WenLan: Bridging Vision and Language by Large-Scale Multi-Modal Pre-Training
Yuqi Huo, Manli Zhang, Guangzhen Liu +32
Multi-modal pre-training models have been intensively explored to bridge vision and language in recent years. However, most of them explicitly model the cross-modal interaction bet…
Contrastive Prototype Learning with Augmented Embeddings for Few-Shot Learning
Yizhao Gao, Nanyi Fei, Guangzhen Liu +3
Most recent few-shot learning (FSL) methods are based on meta-learning with episodic training. In each meta-training episode, a discriminative feature embedding and/or classifier a…
Margin-Based Transfer Bounds for Meta Learning with Deep Feature Embedding
Jiechao Guan, Zhiwu Lu, Tao Xiang +1
By transferring knowledge learned from seen/previous tasks, meta learning aims to generalize well to unseen/future tasks. Existing meta-learning approaches have shown promising emp…
Counterfactual VQA: A Cause-Effect Look at Language Bias
Yulei Niu, Kaihua Tang, Hanwang Zhang +3
VQA models may tend to rely on language bias as a shortcut and thus fail to sufficiently learn the multi-modal knowledge from both vision and language. Recent debiasing methods pro…
Domain-Adaptive Few-Shot Learning
An Zhao, Mingyu Ding, Zhiwu Lu +5
Existing few-shot learning (FSL) methods make the implicit assumption that the few target class samples are from the same domain as the source class samples. However, in practice t…
AdarGCN: Adaptive Aggregation GCN for Few-Shot Learning
Jianhong Zhang, Manli Zhang, Zhiwu Lu +2
Existing few-shot learning (FSL) methods assume that there exist sufficient training samples from source classes for knowledge transfer to target classes with few training samples.…
Meta-Learning across Meta-Tasks for Few-Shot Learning
Nanyi Fei, Zhiwu Lu, Yizhao Gao +3
Existing meta-learning based few-shot learning (FSL) methods typically adopt an episodic training strategy whereby each episode contains a meta-task. Across episodes, these tasks a…
Few-Shot Learning as Domain Adaptation: Algorithm and Analysis
Jiechao Guan, Zhiwu Lu, Tao Xiang +1
To recognize the unseen classes with only few samples, few-shot learning (FSL) uses prior knowledge learned from the seen classes. A major challenge for FSL is that the distributio…
Learning Depth-Guided Convolutions for Monocular 3D Object Detection
Mingyu Ding, Yuqi Huo, Hongwei Yi +4
3D object detection from a single image without LiDAR is a challenging task due to the lack of accurate depth information. Conventional 2D convolutions are unsuitable for this task…
Every Frame Counts: Joint Learning of Video Segmentation and Optical Flow
Mingyu Ding, Zhe Wang, Bolei Zhou +3
A major challenge for video semantic segmentation is the lack of labeled data. In most benchmark datasets, only one frame of a video clip is annotated, which makes most supervised…
Mobile Video Action Recognition
Yuqi Huo, Xiaoli Xu, Yao Lu +3
Video action recognition, which is topical in computer vision and video analysis, aims to allocate a short video clip to a pre-defined category such as brushing hair or climbing st…