29 citations · 130 across the 17 of their papers we have counts for
Showing 2021Show all
3 papers · 1 filter
cs.CL2021★ 27 cited
Taming Sparsely Activated Transformer with Stochastic Experts
Simiao Zuo, Xiaodong Liu, Jian Jiao +5
Sparsely activated models (SAMs), such as Mixture-of-Experts (MoE), can easily scale to have outrageously large amounts of parameters without significant increase in computational…
cs.CL2021★ 3 cited
KFCNet: Knowledge Filtering and Contrastive Learning Network for Generative Commonsense Reasoning
Haonan Li, Yeyun Gong, Jian Jiao +3
Pre-trained language models have led to substantial gains over a broad range of natural language processing (NLP) tasks, but have been shown to have limitations for natural languag…
cs.CL2021★ 7 cited
Mask Attention Networks: Rethinking and Strengthen Transformer
Zhihao Fan, Yeyun Gong, Dayiheng Liu +6
Transformer is an attention-based neural network, which consists of two sublayers, namely, Self-Attention Network (SAN) and Feed-Forward Network (FFN). Existing research explores t…