Exploiting Student Parallelism for Efficient GPU Inference of BERT-like Models in Online Services

Weiyan Wang, Yilun Jin, Yiming Zhang +7

Due to high accuracy, BERT-like models have been widely adopted by text mining and web searching. However, large BERT-like models suffer from inefficient online inference, facing t…