Speeding up Deep Model Training by Sharing Weights and Then Unsharing

Shuo Yang, Le Hou, Xiaodan Song +2

We propose a simple and efficient approach for training the BERT model. Our approach exploits the special structure of BERT that contains a stack of repeated modules (i.e., transfo…