Self-Influence Guided Data Reweighting for Language Model Pre-training

Megh Thakkar, Tolga Bolukbasi, Sriram Ganapathy +3

Language Models (LMs) pre-trained with self-supervision on large text corpora have become the default starting point for developing models for various NLP tasks. Once the pre-train…