416 citations · 3k across the 127 of their papers we have counts for
24 papers · 1 filter
NÜWA: Visual Synthesis Pre-training for Neural visUal World creAtion
Chenfei Wu, Jian Liang, Lei Ji +4
This paper presents a unified multimodal pre-trained model called NÜWA that can generate new or manipulate existing visual data (i.e., images and videos) for various visual synthes…
Adversarial Retriever-Ranker for dense text retrieval
Hang Zhang, Yeyun Gong, Yelong Shen +3
Current dense text retrieval models face two typical challenges. First, they adopt a siamese dual-encoder architecture to encode queries and documents independently for fast indexi…
Long-Range Modeling of Source Code Files with eWASH: Extended Window Access by Syntax Hierarchy
Colin B. Clement, Shuai Lu, Xiaoyu Liu +5
Statistical language modeling and translation with transformers have found many successful applications in program understanding and generation tasks, setting high benchmarks for t…
KFCNet: Knowledge Filtering and Contrastive Learning Network for Generative Commonsense Reasoning
Haonan Li, Yeyun Gong, Jian Jiao +3
Pre-trained language models have led to substantial gains over a broad range of natural language processing (NLP) tasks, but have been shown to have limitations for natural languag…
Contextual Fine-to-Coarse Distillation for Coarse-grained Response Selection in Open-Domain Conversations
Wei Chen, Yeyun Gong, Can Xu +9
We study the problem of coarse-grained response selection in retrieval-based dialogue systems. The problem is equally important with fine-grained response selection, but is less ex…
XLM-K: Improving Cross-Lingual Language Model Pre-training with Multilingual Knowledge
Xiaoze Jiang, Yaobo Liang, Weizhu Chen +1
Cross-lingual pre-training has achieved great successes using monolingual and bilingual plain text corpora. However, most pre-trained models neglect multilingual knowledge, which i…
KD-VLP: Improving End-to-End Vision-and-Language Pretraining with Object Knowledge Distillation
Yongfei Liu, Chenfei Wu, Shao-yen Tseng +3
Self-supervised vision-and-language pretraining (VLP) aims to learn transferable multi-modal representations from large-scale image-text data and to achieve strong performances on…
Discovering Representation Sprachbund For Multilingual Pre-Training
Yimin Fan, Yaobo Liang, Alexandre Muzio +4
Multilingual pre-trained models have demonstrated their effectiveness in many multilingual NLP tasks and enabled zero-shot or few-shot transfer from high-resource languages to low…
Hybrid Reasoning Network for Video-based Commonsense Captioning
Weijiang Yu, Jian Liang, Lei Ji +4
The task of video-based commonsense captioning aims to generate event-wise captions and meanwhile provide multiple commonsense descriptions (e.g., attribute, effect and intention)…
From LSAT: The Progress and Challenges of Complex Reasoning
Siyuan Wang, Zhongkun Liu, Wanjun Zhong +4
Complex reasoning aims to draw a correct inference based on complex rules. As a hallmark of human intelligence, it involves a degree of explicit reading comprehension, interpretati…
FastSeq: Make Sequence Generation Faster
Yu Yan, Fei Hu, Jiusheng Chen +7
Transformer-based models have made tremendous impacts in natural language generation. However the inference speed is a bottleneck due to large model size and intensive computing in…
GEM: A General Evaluation Benchmark for Multimodal Tasks
Lin Su, Nan Duan, Edward Cui +7
In this paper, we present GEM as a General Evaluation benchmark for Multimodal tasks. Different from existing datasets such as GLUE, SuperGLUE, XGLUE and XTREME that mainly focus o…
EL-Attention: Memory Efficient Lossless Attention for Generation
Yu Yan, Jiusheng Chen, Weizhen Qi +4
Transformer model with multi-head attention requires caching intermediate results for efficient inference in generation tasks. However, cache brings new memory-related costs and pr…
Learning to Complete Code with Sketches
Daya Guo, Alexey Svyatkovskiy, Jian Yin +3
Code completion is usually cast as a language modelling problem, i.e., continuing an input in a left-to-right fashion. However, in practice, some parts of the completion (e.g., str…
CoSQA: 20,000+ Web Queries for Code Search and Question Answering
Junjie Huang, Duyu Tang, Linjun Shou +5
Finding codes given natural language query isb eneficial to the productivity of software developers. Future progress towards better semantic matching between query and code require…
Logic-Driven Context Extension and Data Augmentation for Logical Reasoning of Text
Siyuan Wang, Wanjun Zhong, Duyu Tang +5
Logical reasoning of text requires understanding critical logical information in the text and performing inference over them. Large-scale pre-trained models for logical reasoning m…
CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval
Huaishao Luo, Lei Ji, Ming Zhong +4
Video-text retrieval plays an essential role in multi-modal research and has been widely used in many real-world web applications. The CLIP (Contrastive Language-Image Pre-training…
Poolingformer: Long Document Modeling with Pooling Attention
Hang Zhang, Yeyun Gong, Yelong Shen +4
In this paper, we introduce a two-level attention schema, Poolingformer, for long document modeling. Its first level uses a smaller sliding window pattern to aggregate information…
GODIVA: Generating Open-DomaIn Videos from nAtural Descriptions
Chenfei Wu, Lun Huang, Qianxi Zhang +5
Generating videos from text is a challenging task due to its high computational requirements for training and infinite possible answers for evaluation. Existing works typically exp…
AR-LSAT: Investigating Analytical Reasoning of Text
Wanjun Zhong, Siyuan Wang, Duyu Tang +6
Analytical reasoning is an essential and challenging task that requires a system to analyze a scenario involving a set of particular circumstances and perform reasoning over it to…
WhiteningBERT: An Easy Unsupervised Sentence Embedding Approach
Junjie Huang, Duyu Tang, Wanjun Zhong +5
Producing the embedding of a sentence in an unsupervised way is valuable to natural language matching and retrieval problems in practice. In this work, we conduct a thorough examin…
ProphetNet-X: Large-Scale Pre-training Models for English, Chinese, Multi-lingual, Dialog, and Code Generation
Weizhen Qi, Yeyun Gong, Yu Yan +9
Now, the pre-training technique is ubiquitous in natural language processing field. ProphetNet is a pre-training based natural language generation method which shows powerful perfo…
Mask Attention Networks: Rethinking and Strengthen Transformer
Zhihao Fan, Yeyun Gong, Dayiheng Liu +6
Transformer is an attention-based neural network, which consists of two sublayers, namely, Self-Attention Network (SAN) and Feed-Forward Network (FFN). Existing research explores t…
CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation
Shuai Lu, Daya Guo, Shuo Ren +19
Benchmark datasets have a significant impact on accelerating research in programming language tasks. In this paper, we introduce CodeXGLUE, a benchmark dataset to foster machine le…