189 citations · 745 across the 24 of their papers we have counts for
Showing 2020Show all
2 papers · 1 filter
cs.LG2020★ 14 cited
PMI-Masking: Principled masking of correlated spans
Yoav Levine, Barak Lenz, Opher Lieber +4
Masking tokens uniformly at random constitutes a common flaw in the pretraining of Masked Language Models (MLMs) such as BERT. We show that such uniform masking allows an MLM to mi…
cs.CL2020★ 114 cited
The Cost of Training NLP Models: A Concise Overview
Or Sharir, Barak Peleg, Yoav Shoham
We review the cost of training large-scale language models, and the drivers of these costs. The intended audience includes engineers and scientists budgeting their model-training e…