117 citations · 241 across the 11 of their papers we have counts for
Showing 2021Show all
3 papers · 1 filter
cs.CL2021★ 23 cited
Multi-head or Single-head? An Empirical Comparison for Transformer Training
Liyuan Liu, Jialu Liu, Jiawei Han
Multi-head attention plays a crucial role in the recent success of Transformer models, which leads to consistent performance improvements over conventional attention in various app…
cs.CL2021★ 15 cited
NewsEmbed: Modeling News through Pre-trained Document Representations
Jialu Liu, Tianqi Liu, Cong Yu
Effectively modeling text-rich fresh content such as news articles at document-level is a challenging problem. To ensure a content-based model generalize well to a broad range of a…
cs.CL2021★ 1 cited
Training ELECTRA Augmented with Multi-word Selection
Jiaming Shen, Jialu Liu, Tianqi Liu +2
Pre-trained text encoders such as BERT and its variants have recently achieved state-of-the-art performances on many NLP tasks. While being effective, these pre-training methods ty…