1.8k citations · 5.3k across the 25 of their papers we have counts for
10 papers · 1 filter
Unsupervised Parallel Corpus Mining on Web Data
Guokun Lai, Zihang Dai, Yiming Yang
With a large amount of parallel data, neural machine translation systems are able to deliver human-level performance for sentence-level translation. However, it is costly to label…
A Mutual Information Maximization Perspective of Language Representation Learning
Lingpeng Kong, Cyprien de Masson d'Autume, Wang Ling +3
We show state-of-the-art word representation learning methods maximize an objective function that is a lower bound on the mutual information between different parts of a word seque…
XLNet: Generalized Autoregressive Pretraining for Language Understanding
Zhilin Yang, Zihang Dai, Yiming Yang +3
With the capability of modeling bidirectional contexts, denoising autoencoding based pretraining like BERT achieves better performance than pretraining approaches based on autoregr…
Fast and Simple Mixture of Softmaxes with BPE and Hybrid-LightRNN for Language Generation
Xiang Kong, Qizhe Xie, Zihang Dai +1
Mixture of Softmaxes (MoS) has been shown to be effective at addressing the expressiveness limitation of Softmax-based models. Despite the known advantage, MoS is practically seale…
SwitchOut: an Efficient Data Augmentation Algorithm for Neural Machine Translation
Xinyi Wang, Hieu Pham, Zihang Dai +1
In this work, we examine methods for data augmentation for text-based tasks such as neural machine translation (NMT). We formulate the design of a data augmentation policy with des…
From Credit Assignment to Entropy Regularization: Two New Algorithms for Neural Sequence Prediction
Zihang Dai, Qizhe Xie, Eduard Hovy
In this work, we study the credit assignment problem in reward augmented maximum likelihood (RAML) learning, and establish a theoretical equivalence between the token-level counter…
Breaking the Softmax Bottleneck: A High-Rank RNN Language Model
Zhilin Yang, Zihang Dai, Ruslan Salakhutdinov +1
We formulate language modeling as a matrix factorization problem, and show that the expressiveness of Softmax-based models (including the majority of neural language models) is lim…
Large-scale Cloze Test Dataset Created by Teachers
Qizhe Xie, Guokun Lai, Zihang Dai +1
Cloze tests are widely adopted in language exams to evaluate students' language proficiency. In this paper, we propose the first large-scale human-created cloze test dataset CLOTH,…
An Interpretable Knowledge Transfer Model for Knowledge Base Completion
Qizhe Xie, Xuezhe Ma, Zihang Dai +1
Knowledge bases are important resources for a variety of natural language processing tasks but suffer from incompleteness. We propose a novel embedding model, \emph{ITransF}, to pe…
CFO: Conditional Focused Neural Question Answering with Large-scale Knowledge Bases
Zihang Dai, Lei Li, Wei Xu
How can we enable computers to automatically answer questions like "Who created the character Harry Potter"? Carefully built knowledge bases provide rich sources of facts. However,…