607 citations · 914 across the 15 of their papers we have counts for
15 papers
Overfitting or Underfitting? Understand Robustness Drop in Adversarial Training
Zichao Li, Liyuan Liu, Chengyu Dong +1
Our goal is to understand why the robustness drops after conducting adversarial training for too long. Although this phenomenon is commonly explained as overfitting, our analysis s…
On the Transformer Growth for Progressive BERT Training
Xiaotao Gu, Liyuan Liu, Hongkun Yu +3
Due to the excessive cost of large-scale language model pre-training, considerable efforts have been made to train BERT progressively -- start from an inferior but low-cost model a…
Very Deep Transformers for Neural Machine Translation
Xiaodong Liu, Kevin Duh, Liyuan Liu +1
We explore the application of very deep Transformer models for Neural Machine Translation (NMT). Using a simple yet effective initialization technique that stabilizes training, we…
Partially-Typed NER Datasets Integration: Connecting Practice to Theory
Shi Zhi, Liyuan Liu, Yu Zhang +4
While typical named entity recognition (NER) models require the training set to be annotated with all target types, each available datasets may only cover a part of them. Instead o…
Understanding the Difficulty of Training Transformers
Liyuan Liu, Xiaodong Liu, Jianfeng Gao +2
Transformers have proved effective in many NLP tasks. However, their training requires non-trivial efforts regarding designing cutting-edge optimizers and learning rate schedulers…
Learning to Contextually Aggregate Multi-Source Supervision for Sequence Labeling
Ouyu Lan, Xiao Huang, Bill Yuchen Lin +3
Sequence labeling is a fundamental framework for various natural language processing problems. Its performance is largely influenced by the annotation quality and quantity in super…
CrossWeigh: Training Named Entity Tagger from Imperfect Annotations
Zihan Wang, Jingbo Shang, Liyuan Liu +3
Everyone makes mistakes. So do human annotators when curating labels for named entity recognition (NER). Such label mistakes might hurt model training and interfere model compariso…
Facet-Aware Evaluation for Extractive Summarization
Yuning Mao, Liyuan Liu, Qi Zhu +2
Commonly adopted metrics for extractive summarization focus on lexical overlap at the token level. In this paper, we present a facet-aware evaluation setup for better assessment of…
On the Variance of the Adaptive Learning Rate and Beyond
Liyuan Liu, Haoming Jiang, Pengcheng He +4
The learning rate warmup heuristic achieves remarkable success in stabilizing training, accelerating convergence and improving generalization for adaptive stochastic optimization a…
Cross-relation Cross-bag Attention for Distantly-supervised Relation Extraction
Yujin Yuan, Liyuan Liu, Siliang Tang +5
Distant supervision leverages knowledge bases to automatically label instances, thus allowing us to train relation extractor without human annotations. However, the generated train…
Learning Named Entity Tagger using Domain-Specific Dictionary
Jingbo Shang, Liyuan Liu, Xiang Ren +3
Recent advances in deep neural models allow us to build reliable named entity recognition (NER) systems without handcrafting features. However, such methods require large amounts o…
Efficient Contextualized Representation: Language Model Pruning for Sequence Labeling
Liyuan Liu, Xiang Ren, Jingbo Shang +2
Many efforts have been made to facilitate natural language processing tasks with pre-trained language models (LMs), and brought significant improvements to various applications. To…
Wikidata Vandalism Detection - The Loganberry Vandalism Detector at WSDM Cup 2017
Qi Zhu, Hongwei Ng, Liyuan Liu +4
Wikidata is the new, large-scale knowledge base of the Wikimedia Foundation. As it can be edited by anyone, entries frequently get vandalized, leading to the possibility that it mi…
Empower Sequence Labeling with Task-Aware Neural Language Model
Liyuan Liu, Jingbo Shang, Frank F. Xu +4
Linguistic sequence labeling is a general modeling approach that encompasses a variety of problems, such as part-of-speech tagging and named entity recognition. Recent advances in…
Heterogeneous Supervision for Relation Extraction: A Representation Learning Approach
Liyuan Liu, Xiang Ren, Qi Zhu +4
Relation extraction is a fundamental task in information extraction. Most existing methods have heavy reliance on annotations labeled by human experts, which are costly and time-co…