LV-BERT: Exploiting Layer Variety for BERT

Weihao Yu, Zihang Jiang, Fei Chen +2

Modern pre-trained language models are mostly built upon backbones stacking self-attention and feed-forward layers in an interleaved order. In this paper, beyond this stereotyped l…