Drop Dropout on Single-Epoch Language Model Pretraining

Houjun Liu, John Bauer, Christopher D. Manning

Originally, dropout was seen as a breakthrough regularization technique that reduced overfitting and improved performance in almost all applications of deep learning by reducing ov…