Domain-oriented Language Pre-training with Adaptive Hybrid Masking and Optimal Transport Alignment

Denghui Zhang, Zixuan Yuan, Yanchi Liu +4

Motivated by the success of pre-trained language models such as BERT in a broad range of natural language processing (NLP) tasks, recent research efforts have been made for adaptin…