Extremely Small BERT Models from Mixed-Vocabulary Training

Sanqiang Zhao, Raghav Gupta, Yang Song +1

Pretrained language models like BERT have achieved good results on NLP tasks, but are impractical on resource-limited devices due to memory footprint. A large fraction of this foot…