ZEN: Pre-training Chinese Text Encoder Enhanced by N-gram Representations

Shizhe Diao, Jiaxin Bai, Yan Song +2

The pre-training of text encoders normally processes text as a sequence of tokens corresponding to small text units, such as word pieces in English and characters in Chinese. It om…