Speeding up Deep Model Training by Sharing Weights and Then Unsharing

Abstract

We propose a simple and efficient approach for training the BERT model. Our approach exploits the special structure of BERT that contains a stack of repeated modules (i.e., transformer encoders). Our proposed approach first trains BERT with the weights shared across all the repeated modules till some point. This is for learning the commonly shared component of weights across all repeated layers. We then stop weight sharing and continue training until convergence. We present theoretic insights for training by sharing weights then unsharing with analysis for simplified models. Empirical experiments on the BERT model show that our method yields better performance of trained models, and significantly reduces the number of training iterations.

References in corpus (10)

• Decoupled Weight Decay Regularization
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Language Models are Few-Shot Learners
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
• Deep Equilibrium Models
• On Layer Normalization in the Transformer Architecture
• A Convergence Analysis of Gradient Descent for Deep Linear Neural Networks
• Batch Normalization Biases Residual Blocks Towards the Identity Function in Deep Networks
• Deeper Insights into Weight Sharing in Neural Architecture Search