Effective Unsupervised Domain Adaptation with Adversarially Trained Language Models

Abstract

Recent work has shown the importance of adaptation of broad-coverage contextualised embedding models on the domain of the target task of interest. Current self-supervised adaptation methods are simplistic, as the training signal comes from a small percentage of \emph{randomly} masked-out tokens. In this paper, we show that careful masking strategies can bridge the knowledge gap of masked language models (MLMs) about the domains more effectively by allocating self-supervision where it is needed. Furthermore, we propose an effective training strategy by adversarially masking out those tokens which are harder to reconstruct by the underlying MLM. The adversarial objective leads to a challenging combinatorial optimisation problem over \emph{subsets} of tokens, which we tackle efficiently through relaxation to a variational lowerbound and dynamic programming. On six unsupervised domain adaptation tasks involving named entity recognition, our method strongly outperforms the random masking strategy and achieves up to +1.64 F1 score improvements.

arXiv commentEMNLP2020

References in corpus (9)

• BioBERT: a pre-trained biomedical language representation model for biomedical text mining
• Categorical Reparameterization with Gumbel-Softmax
• XLNet: Generalized Autoregressive Pretraining for Language Understanding
• GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
• Wasserstein Distance Guided Representation Learning for Domain Adaptation
• Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Tasks
• Adversarial Training for Large Neural Language Models
• Improving Neural Language Modeling via Adversarial Training
• Learning Household Task Knowledge from WikiHow Descriptions