84 citations · 308 across the 21 of their papers we have counts for
21 papers
Healing Unsafe Dialogue Responses with Weak Supervision Signals
Zi Liang, Pinghui Wang, Ruofei Zhang +3
Recent years have seen increasing concerns about the unsafe response generation of large-scale dialogue systems, where agents will learn offensive or biased behaviors from the real…
MERGE: Fast Private Text Generation
Zi Liang, Pinghui Wang, Ruofei Zhang +3
The drastic increase in language models' parameters has led to a new trend of deploying models in cloud servers, raising growing concerns about private inference for Transformer-ba…
SwiftPruner: Reinforced Evolutionary Pruning for Efficient Ad Relevance
Li Lyna Zhang, Youkow Homma, Yujing Wang +5
Ad relevance modeling plays a critical role in online advertising systems including Microsoft Bing. To leverage powerful transformers like BERT in this low-latency setting, many ex…
A Self-Paced Mixed Distillation Method for Non-Autoregressive Generation
Weizhen Qi, Yeyun Gong, Yelong Shen +6
Non-Autoregressive generation is a sequence generation paradigm, which removes the dependency between target tokens. It could efficiently reduce the text generation latency with pa…
Taming Sparsely Activated Transformer with Stochastic Experts
Simiao Zuo, Xiaodong Liu, Jian Jiao +5
Sparsely activated models (SAMs), such as Mixture-of-Experts (MoE), can easily scale to have outrageously large amounts of parameters without significant increase in computational…
KFCNet: Knowledge Filtering and Contrastive Learning Network for Generative Commonsense Reasoning
Haonan Li, Yeyun Gong, Jian Jiao +3
Pre-trained language models have led to substantial gains over a broad range of natural language processing (NLP) tasks, but have been shown to have limitations for natural languag…
EL-Attention: Memory Efficient Lossless Attention for Generation
Yu Yan, Jiusheng Chen, Weizhen Qi +4
Transformer model with multi-head attention requires caching intermediate results for efficient inference in generation tasks. However, cache brings new memory-related costs and pr…
ProphetNet-X: Large-Scale Pre-training Models for English, Chinese, Multi-lingual, Dialog, and Code Generation
Weizhen Qi, Yeyun Gong, Yu Yan +9
Now, the pre-training technique is ubiquitous in natural language processing field. ProphetNet is a pre-training based natural language generation method which shows powerful perfo…
Mask Attention Networks: Rethinking and Strengthen Transformer
Zhihao Fan, Yeyun Gong, Dayiheng Liu +6
Transformer is an attention-based neural network, which consists of two sublayers, namely, Self-Attention Network (SAN) and Feed-Forward Network (FFN). Existing research explores t…
TextGNN: Improving Text Encoder via Graph Neural Network in Sponsored Search
Jason Yue Zhu, Yanling Cui, Yuming Liu +7
Text encoders based on C-DSSM or transformers have demonstrated strong performance in many Natural Language Processing (NLP) tasks. Low latency variants of these models have also b…
BANG: Bridging Autoregressive and Non-autoregressive Generation with Large Scale Pretraining
Weizhen Qi, Yeyun Gong, Jian Jiao +9
In this paper, we propose BANG, a new pretraining model to Bridge the gap between Autoregressive (AR) and Non-autoregressive (NAR) Generation. AR and NAR generation can be uniforml…
An Enhanced Knowledge Injection Model for Commonsense Generation
Zhihao Fan, Yeyun Gong, Zhongyu Wei +6
Commonsense generation aims at generating plausible everyday scenario description based on a set of provided concepts. Digging the relationship of concepts from scratch is non-triv…
GLGE: A New General Language Generation Evaluation Benchmark
Dayiheng Liu, Yu Yan, Yeyun Gong +15
Multi-task benchmarks such as GLUE and SuperGLUE have driven great progress of pretraining and transfer learning in Natural Language Processing (NLP). These benchmarks mostly focus…
ProphetNet-Ads: A Looking Ahead Strategy for Generative Retrieval Models in Sponsored Search Engine
Weizhen Qi, Yeyun Gong, Yu Yan +6
In a sponsored search engine, generative retrieval models are recently proposed to mine relevant advertisement keywords for users' input queries. Generative retrieval models genera…
AutoADR: Automatic Model Design for Ad Relevance
Yiren Chen, Yaming Yang, Hong Sun +7
Large-scale pre-trained models have attracted extensive attention in the research community and shown promising results on various tasks of natural language processing. However, th…
HittER: Hierarchical Transformers for Knowledge Graph Embeddings
Sanxing Chen, Xiaodong Liu, Jianfeng Gao +3
This paper examines the challenging problem of learning representations of entities and relations in a complex multi-relational knowledge graph. We propose HittER, a Hierarchical T…
TwinBERT: Distilling Knowledge to Twin-Structured BERT Models for Efficient Retrieval
Wenhao Lu, Jian Jiao, Ruofei Zhang
Pre-trained language models like BERT have achieved great success in a wide variety of NLP tasks, while the superior performance comes with high demand in computational resources,…
ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-training
Weizhen Qi, Yu Yan, Yeyun Gong +5
This paper presents a new sequence-to-sequence pre-training model called ProphetNet, which introduces a novel self-supervised objective named future n-gram prediction and the propo…
DeepProbe: Information Directed Sequence Understanding and Chatbot Design via Recurrent Neural Networks
Zi Yin, Keng-hao Chang, Ruofei Zhang
Information extraction and user intention identification are central topics in modern query understanding and recommendation systems. In this paper, we propose DeepProbe, a generic…
Large-Scale Multi-Label Learning with Incomplete Label Assignments
Xiangnan Kong, Zhaoming Wu, Li-Jia Li +4
Multi-label learning deals with the classification problems where each instance can be assigned with multiple labels simultaneously. Conventional multi-label learning approaches ma…
A Lipschitz Exploration-Exploitation Scheme for Bayesian Optimization
Ali Jalali, Javad Azimi, Xiaoli Fern +1
The problem of optimizing unknown costly-to-evaluate functions has been studied for a long time in the context of Bayesian Optimization. Algorithms in this field aim to find the op…