LV-BERT: Exploiting Layer Variety for BERT

Abstract

Modern pre-trained language models are mostly built upon backbones stacking self-attention and feed-forward layers in an interleaved order. In this paper, beyond this stereotyped layer pattern, we aim to improve pre-trained models by exploiting layer variety from two aspects: the layer type set and the layer order. Specifically, besides the original self-attention and feed-forward layers, we introduce convolution into the layer type set, which is experimentally found beneficial to pre-trained models. Furthermore, beyond the original interleaved order, we explore more layer orders to discover more powerful architectures. However, the introduced layer variety leads to a large architecture space of more than billions of candidates, while training a single candidate model from scratch already requires huge computation cost, making it not affordable to search such a space by directly training large amounts of candidate models. To solve this problem, we first pre-train a supernet from which the weights of all candidate models can be inherited, and then adopt an evolutionary algorithm guided by pre-training accuracy to find the optimal architecture. Extensive experiments show that LV-BERT model obtained by our method outperforms BERT and its variants on various downstream tasks. For example, LV-BERT-small achieves 79.8 on the GLUE testing set, 1.8 higher than the strong baseline ELECTRA-small.

arXiv commentAccepted to Findings of ACL 2021. The code and pre-trained models are available at https://github.com/yuweihao/LV-BERT

References in corpus (21)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Neural Architecture Search with Reinforcement Learning
• Language Models are Few-Shot Learners
• XLNet: Generalized Autoregressive Pretraining for Language Understanding
• DARTS: Differentiable Architecture Search
• SQuAD: 100,000+ Questions for Machine Comprehension of Text
• Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books
• ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware
• SNAS: Stochastic Neural Architecture Search
• Single Path One-Shot Neural Architecture Search with Uniform Sampling
• Learned in Translation: Contextualized Word Vectors
• HAT: Hardware-Aware Transformers for Efficient Natural Language Processing
• Know What You Don't Know: Unanswerable Questions for SQuAD
• A Comprehensive Survey of Neural Architecture Search: Challenges and Solutions
• DetNAS: Backbone Search for Object Detection
• Auto-DeepLab: Hierarchical Neural Architecture Search for Semantic Image Segmentation
• ConvBERT: Improving BERT with Span-based Dynamic Convolution
• Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks
• Multi-node Bert-pretraining: Cost-efficient Approach
• AutoRC: Improving BERT Based Relation Classification Models via Architecture Search