Pre-training Language Model Incorporating Domain-specific Heterogeneous Knowledge into A Unified Representation

Hongyin Zhu, Hao Peng, Zhiheng Lyu +3

Existing technologies expand BERT from different perspectives, e.g. designing different pre-training tasks, different semantic granularities, and different model architectures. Few…