Multi-node Bert-pretraining: Cost-efficient Approach

Jiahuang Lin, Xin Li, Gennady Pekhimenko

Recently, large scale Transformer-based language models such as BERT, GPT-2, and XLNet have brought about exciting leaps in state-of-the-art results for many Natural Language Proce…