Distilling BERT for low complexity network training

Bansidhar Mangalwedhekar

This paper studies the efficiency of transferring BERT learnings to low complexity models like BiLSTM, BiLSTM with attention and shallow CNNs using sentiment analysis on SST-2 data…