Large-Scale Differentially Private BERT

Rohan Anil, Badih Ghazi, Vineet Gupta +2

In this work, we study the large-scale pretraining of BERT-Large with differentially private SGD (DP-SGD). We show that combined with a careful implementation, scaling up the batch…