941 citations · 2.8k across the 32 of their papers we have counts for
5 papers · 1 filter
Distilled Dual-Encoder Model for Vision-Language Understanding
Zekun Wang, Wenhui Wang, Haichao Zhu +3
We propose a cross-modal attention distillation framework to train a dual-encoder model for vision-language understanding tasks, such as visual reasoning and visual question answer…
VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts
Hangbo Bao, Wenhui Wang, Li Dong +5
We present a unified Vision-Language pretrained Model (VLMo) that jointly learns a dual encoder and a fusion encoder with a modular Transformer network. Specifically, we introduce…
s2s-ft: Fine-Tuning Pretrained Transformer Encoders for Sequence-to-Sequence Learning
Hangbo Bao, Li Dong, Wenhui Wang +2
Pretrained bidirectional Transformers, such as BERT, have achieved significant improvements in a wide variety of language understanding tasks, while it is not straightforward to di…
Adapt-and-Distill: Developing Small, Fast and Effective Pretrained Language Models for Domains
Yunzhi Yao, Shaohan Huang, Wenhui Wang +2
Large pre-trained models have achieved great success in many natural language processing tasks. However, when they are applied in specific domains, these models suffer from domain…
Consistency Regularization for Cross-Lingual Fine-Tuning
Bo Zheng, Li Dong, Shaohan Huang +7
Fine-tuning pre-trained cross-lingual language models can transfer task-specific supervision from one language to the others. In this work, we propose to improve cross-lingual fine…