607 citations · 914 across the 15 of their papers we have counts for
4 papers · 1 filter
Overfitting or Underfitting? Understand Robustness Drop in Adversarial Training
Zichao Li, Liyuan Liu, Chengyu Dong +1
Our goal is to understand why the robustness drops after conducting adversarial training for too long. Although this phenomenon is commonly explained as overfitting, our analysis s…
Partially-Typed NER Datasets Integration: Connecting Practice to Theory
Shi Zhi, Liyuan Liu, Yu Zhang +4
While typical named entity recognition (NER) models require the training set to be annotated with all target types, each available datasets may only cover a part of them. Instead o…
Understanding the Difficulty of Training Transformers
Liyuan Liu, Xiaodong Liu, Jianfeng Gao +2
Transformers have proved effective in many NLP tasks. However, their training requires non-trivial efforts regarding designing cutting-edge optimizers and learning rate schedulers…
On the Variance of the Adaptive Learning Rate and Beyond
Liyuan Liu, Haoming Jiang, Pengcheng He +4
The learning rate warmup heuristic achieves remarkable success in stabilizing training, accelerating convergence and improving generalization for adaptive stochastic optimization a…